Libx Awesomeリスト

    このページの内容

    Awesome DATA SCIENCE

    DATA SCIENCEを扱う資料や関連プロジェクトをまとめたAwesomeリストです。

    Table of Contents

    What is Data Science?

    ^ back to top ^

    データサイエンスは、現在のコンピュータとインターネットの農場で最も注目されているテーマの一つです。人々はアプリケーションやシステムからデータを収集してきたのです。今こそ、それらのデータを分析する時期です。次に進むべきステップは、データから提案を生み出し、未来についての予測を作成することです。 Here で、データサイエンスに関する最大の問いと、専門家からの数百の回答を見つけることができます。

    LinkPreview
    Data Science For BeginnersMicrosoft are pleased to offer a 10-week, 20-lesson curriculum all about Data Science.
    What is Data Science @ O’reillyData scientists combine entrepreneurship with patience, the willingness to build data products incrementally, the ability to explore, and the ability to iterate over a solution. They are inherently interdisciplinary. They can tackle all aspects of a problem, from initial data collection and data conditioning to drawing conclusions. They can think outside the box to come up with new ways to view the problem, or to work with very broadly defined problems: “here’s a lot of data, what can you make from it?”
    What is Data Science @ QuoraData Science is a combination of a number of aspects of Data such as Technology, Algorithm development, and data interference to study the data, analyse it, and find innovative solutions to difficult problems. Basically Data Science is all about Analysing data and driving for business growth by finding creative ways.
    The sexiest job of 21st centuryData scientists today are akin to Wall Street “quants” of the 1980s and 1990s. In those days people with backgrounds in physics and math streamed to investment banks and hedge funds, where they could devise entirely new algorithms and data strategies. Then a variety of universities developed master’s programs in financial engineering, which churned out a second generation of talent that was more accessible to mainstream firms. The pattern was repeated later in the 1990s with search engineers, whose rarefied skills soon came to be taught in computer science programs.
    WikipediaData science is an interdisciplinary field that uses scientific methods, processes, algorithms and systems to extract knowledge and insights from many structural and unstructured data. Data science is related to data mining, machine learning and big data.
    How to Become a Data ScientistData scientists are big data wranglers, gathering and analyzing large sets of structured and unstructured data. A data scientist’s role combines computer science, statistics, and mathematics. They analyze, process, and model data then interpret the results to create actionable plans for companies and other organizations.
    a very short history of #datascienceThe story of how data scientists became sexy is mostly the story of the coupling of the mature discipline of statistics with a very young one—computer science. The term “Data Science” has emerged only recently to specifically designate a new profession that is expected to make sense of the vast stores of big data. But making sense of data has a long history and has been discussed by scientists, statisticians, librarians, computer scientists and others for years. The following timeline traces the evolution of the term “Data Science” and its use, attempts to define it, and related terms.
    Software Development Resources for Data ScientistsData scientists concentrate on making sense of data through exploratory analysis, statistics, and models. Software developers apply a separate set of knowledge with different tools. Although their focus may seem unrelated, data science teams can benefit from adopting software development best practices. Version control, automated testing, and other dev skills help create reproducible, production-ready code and tools.
    Data Scientist RoadmapData science is an excellent career choice in today’s data-driven world where approx 328.77 million terabytes of data are generated daily. And this number is only increasing day by day, which in turn increases the demand for skilled data scientists who can utilize this data to drive business growth.
    Navigating Your Path to Becoming a Data Scientist_Data science is one of the most in-demand careers today. With businesses increasingly relying on data to make decisions, the need for skilled data scientists has grown rapidly. Whether it’s tech companies, healthcare organizations, or even government institutions, data scientists play a crucial role in turning raw data into valuable insights. But how do you become a data scientist, especially if you’re just starting out? _

    Where do I Start?

    ^ back to top ^

    プログラミング言語を持つことは、厳密には必須ではありませんが、データサイエンティストとして効果的に働くために非常に重要なスキルです。現在、最も人気のある言語は Python で、それに次いで R が人気です。Pythonは、さまざまな分野に応用される汎用的なスクリプト言語です。Rは統計に特化した言語で、多くの統計ツールを標準で備えています。

    Pythonは、使用のしやすさやユーザーが生成したパッケージの豊かな生態系のおかげで、科学分野で最も人気のある言語です。パッケージのインストールには、主に2つの方法があります：Pythonが標準で提供するパッケージマネージャーであるPip（呼び出し名：pip install）と、Python、R用にパッケージをインストールできるだけでなく、Gitなどの実行ファイルをダウンロードできる強力なパッケージマネージャーであるAnaconda（呼び出し名：conda install）です。

    Rとは異なり、Pythonはデータサイエンスを設計の中心に置いたものではなく、しかし、その欠如を補うための多くの第三者ライブラリがあります。このドキュメントの後半に、パッケージのより詳細なリストが掲載されていますが、これら4つのパッケージはデータサイエンスの旅を始めるのに適した選択肢です：Scikit-Learnは汎用的なデータサイエンスパッケージであり、最も人気のあるアルゴリズムを実装しています。また、その実装するモデルについての豊かなドキュメンテーション、チュートリアル、および例が含まれています。あなたが自作の実装を好む場合でも、Scikit-Learnは多くの一般的なアルゴリズムの内部構造を理解するための貴重な参考資料です。Pandasを使用することで、データを収集し、分析して便利なテーブル形式に変換できます。Numpyはベクトルと行列に焦点を当てた数学演算の高速ツールを提供しています。Seabornは、Matplotlibパッケージに基づいており、データの美しく視覚的に表現するための迅速な手段であり、多くのデフォルト設定が用意されており、データの多くの一般的な可視化方法を示すギャラリーも提供されています。

    データサイエンティストになる旅を始める際、言語の選択は特に重要ではなく、PythonとRのそれぞれには利点と欠点があります。好きな言語を選んで、下記にリストされたFree coursesのいずれかをチェックしてみてください！

    Beginner Roadmap

    If you’re just starting out, here’s a simple recommended path:

    1. Learn Python – Start with basics: variables, loops, functions
    2. Learn core libraries – Pandas, NumPy, Matplotlib, Scikit-Learn
    3. Practice with beginner projects – Try Titanic survival or house price prediction on Kaggle
    4. Learn Math basics – Statistics, Linear Algebra, Probability
    5. Move into ML – Supervised learning → Unsupervised → Deep Learning

    Agents

    このセクションには、データサイエンスのワークフローに役立つエージェントフレームワークとツールが含まれています。

    Frameworks

    • ADK-Rust - RustでAIアグエントを開発できるプロダクション用開発キット。モデルに依存しない設計（Gemini、OpenAI、Anthropic）、複数のアグエントタイプ（LLM、グラフ、ワークフロー）、MCP対応、内蔵テレメトリ。
    • Lumen - データとのチャットを可能にするアグエントフレームワーク。自然言語をSQLに変換し、変換パイプラインや可視化を実現。出力は宣言型仕様であり、検証・編集・ノートブックに再開・ダッシュボードに組み込むことが可能。

    Tools

    • Frostbyte MCP - AIアグエントが利用できる13のデータツールを提供するMCPサーバー：リアルタイムの暗号通貨価格、IPの地理位置、DNS検索、ウェブスクレイピング（マークダウンへ）、コード実行、スクリーンショット。1つのAPIキーで40以上のサービスにアクセス可能。
    • Arch Tools - データサイエンスワークフロー向けの61のプロダクション用AI APIツール：コード分析、ウェブスクレイピング、NLP、画像生成、暗号資産データ、検索。REST APIおよびMCPプロトコル対応。GitHub
    • Not Human Search - AIアグエントが利用できる9,000以上のAIツールとAPIをインデックスする検索エンジン。各ツールのアグエント対応度を評価（llms.txt、OpenAPI、MCP、ai-plugin.json）。プログラムによるツール発見用のREST APIおよびMCPサーバー。GitHub
    • DeepAlpha - LightGBM + XGBoostのアンサンブルモデルを用いたAI暗号取引フレームワーク。72のML特徴量を活用し、外サンプルデータでの検証精度は70.9％。BybitおよびBinanceに対応。MITライセンス、PyPIで利用可能。
    • CAJAL - 実際のarXiv引用を含む、出版用の科学論文を生成するローカルAIアグエント。IMRaD構造と審査スコアを提供。Ollamaで完全にオフラインで動作し、4B～9Bモデルを用いる。MITライセンス。HuggingFace
    • ai-evaluation - 50以上のメトリクスを備えたオープンソースLLMおよびアグエント評価フレームワーク。LLM-as-Judgeの拡張とガードレールスキャナ（ジャイルブレイク、PII、プロンプトインジェクション）。データサイエンスワークフローにおけるRAG出力、アグエントの行動、関数呼び出しの評価に有効。

    Research & Knowledge Retrieval

    • BGPT MCP - AIアグエントが、全文研究から抽出された原始実験データをもとに構築された科学論文データベースにアクセスできるMCPサーバー。各論文に対して25以上の構造化フィールドを返す（方法、結果、サンプルサイズ、品質スコアなど）。GitHub

    • Chunk Tuner - RAGにおけるドキュメントチャンク戦略のベンチマーク、リトリーブ品質の評価、コーパスの設定を推奨するためのオープンソースPythonライブラリおよびMCPサーバー。

    • II-Commons - arXiv、PubMed/PMC、および対応する米国政策データベースを対象とした、決定論的なリトリーブを実現する毎日のスキルとCLI。

    • Spraay x402 Gateway - x402支払いゲートウェイ。AIアグエント向け23の研究・参考端末：ウィキペディア、arXiv、PubMed、Wikidata、学術引用検索、エンティティ抽出など。BaseおよびSolana上でUSDCでコールごとに支払い。APIキーまたはサブスクリプション不要。さらに、地図、AI推論、DeFi、計算など39カテゴリの150以上の端末を提供。GitHub

    • Suppr - 研究者向けのAI文献検索、ドキュメント翻訳、深層研究ワークスペース。

    Workflow

    ^ back to top ^

    • sim - Sim Studioのインターフェースは、好きなツールと連携するLLMを迅速に構築・デプロイできる軽量かつ直感的なインターフェース。

    Training Resources

    ^ back to top ^

    データサイエンスをどう学ぶか？データサイエンスを実際にやることで、もちろん！まあ、最初の段階ではそれだけではあまり役立たないかもしれません。このセクションでは、学習リソースを、やや順番に、最小のコミットメントから最大のコミットメントまで、 Tutorials、 Massively Open Online Courses (MOOCs)、 Intensive Programs、および Colleges とリストアップしています。

    Tutorials

    ^ back to top ^

    Free Courses

    ^ back to top ^

    • Data Science - オープンソース社会大学
    • Data Scientist with R
    • Data Scientist with Python
    • Genetic Algorithms OCW Course
    • AI Expert Roadmap - 人工知能専門家になるための道筋
    • Convex Optimization - 凸最適化（凸解析の基礎；最小二乗法、線形および二次計画問題、半正定値計画問題、ミニマックス、極端な体積など、その他問題；最適性条件、双対理論など）
    • Learning from Data - 機械学習の基礎理論、アルゴリズムおよび応用についての紹介
    • Kaggle - データサイエンス、機械学習、Pythonなどについて学ぶ
    • ML Observability Fundamentals - 生産環境におけるML問題の監視および原因究明方法を学ぶ
    • Weights & Biases Effective MLOps: Model Development - W&Bを用いてエンドツーエンドな機械を構築するための無料コースと認定
    • Python for Data Science by Scaler - 本コースは、データ駆動型世界で優れたスキルを身につけるために初心者を支援するように設計されています。包括的なカリキュラムにより、統計学、プログラミング、データ可視化、機械学習の基礎をしっかり学ぶことができます。
    • MLSys-NYU-2022 - NYU Tandonにおける2022年の金融機械学習コースのスライド、スクリプト、資料。
    • Hands-on Train and Deploy ML - サーバレスAPIを訓練・デプロイするための実践的なコース。暗号通貨価格を予測する。
    • LLMOps: Building Real-World Applications With Large Language Models - 最新のツールと技術を用いて、LLMを用いた現代的なソフトウェア開発を学ぶ。
    • Prompt Engineering for Vision Models - 自然言語で、ポイント、境界ボックス、セグメンテーションマスク、さらには他の画像まで、最先端のコンピュータビジョンモデルにプロンプトを送る方法を学ぶ。DeepLearning.AIの無料コース。
    • Data Science Course By IBM - データサイエンスの基礎とその業界での応用についての無料リソースを提供。
    • Neural Networks: Zero to Hero - アンデル・カーパティの無料動画シリーズ。ゼロからニューラルネットワークを学ぶ——バックプロパゲーション、makemore、GPTなど。

    MOOC’s

    ^ back to top ^

    Intensive Programs

    ^ back to top ^

    Colleges

    ^ back to top ^

    The Data Science Toolbox

    ^ back to top ^

    このセクションは、データサイエンス世界におけるパッケージ、ツール、アルゴリズム、およびその他の有用なアイテムのコレクションです。

    Algorithms

    ^ back to top ^

    これらは、データを理解し、その中から意味を導くために使える機械学習およびデータマイニングのアルゴリズムとモデルです。

    Three kinds of Machine Learning Systems

    • Based on training with human supervision
    • Based on learning incrementally on fly
    • Based on data points comparison and pattern detection

    Comparison

    • datacompy - DataComPyは、2つのPandasのデータフレームを比較するためのパッケージ。

    Supervised Learning

    Unsupervised Learning

    Semi-Supervised Learning

    Reinforcement Learning

    Data Mining Algorithms

    Modern Data Mining Algorithms

    Deep Learning architectures

    General Machine Learning Packages

    ^ back to top ^

    Deep Learning Packages

    PyTorch Ecosystem

    TensorFlow Ecosystem

    Keras Ecosystem

    Visualization Tools

    ^ back to top ^

    Miscellaneous Tools

    ^ back to top ^

    LinkDescription
    The Data Science Lifecycle ProcessThe Data Science Lifecycle Process is a process for taking data science teams from Idea to Value repeatedly and sustainably. The process is documented in this repo
    Data Science Lifecycle Template RepoTemplate repository for data science lifecycle project
    TabGANSynthetic tabular data generation using GANs, Diffusion Models, and LLMs with adversarial filtering and privacy metrics.
    RexMexA general purpose recommender metrics library for fair evaluation.
    ChemicalXA PyTorch based deep learning library for drug pair scoring.
    FileShot.ioSecure zero-knowledge encrypted file sharing (AES-256-GCM in-browser). No account required, MIT licensed, self-hostable, optional link expiry.
    CorpusExplorerSoftware for corpus linguists and text/data mining enthusiasts. Build your own corpora in over 60 languages. Use over 50 tools/visualizations.
    PyTorch Geometric TemporalRepresentation learning on dynamic graphs.
    Little Ball of FurA graph sampling library for NetworkX with a Scikit-Learn like API.
    Karate ClubAn unsupervised machine learning extension library for NetworkX with a Scikit-Learn like API.
    ML WorkspaceAll-in-one web-based IDE for machine learning and data science. The workspace is deployed as a Docker container and is preloaded with a variety of popular data science libraries (e.g., Tensorflow, PyTorch) and dev tools (e.g., Jupyter, VS Code)
    xonsh shellA Python-powered shell that enables integration, management and orchestration of data science libraries mostly written in Python, allowing you to build pipelines, code and command-based workflows. It can also be used as a kernel for Jupyter Notebook.
    Neptune.aiCommunity-friendly platform supporting data scientists in creating and sharing machine learning models. Neptune facilitates teamwork, infrastructure management, models comparison and reproducibility.
    steppyLightweight, Python library for fast and reproducible machine learning experimentation. Introduces very simple interface that enables clean machine learning pipeline design.
    steppy-toolkitCurated collection of the neural networks, transformers and models that make your machine learning work faster and more effective.
    Datalab from Googleeasily explore, visualize, analyze, and transform data using familiar languages, such as Python and SQL, interactively.
    Hortonworks Sandboxis a personal, portable Hadoop environment that comes with a dozen interactive Hadoop tutorials.
    Ris a free software environment for statistical computing and graphics.
    Tidyverseis an opinionated collection of R packages designed for data science. All packages share an underlying design philosophy, grammar, and data structures.
    RStudioIDE – powerful user interface for R. It’s free and open source, and works on Windows, Mac, and Linux.
    Python - Pandas - AnacondaCompletely free enterprise-ready Python distribution for large-scale data processing, predictive analytics, and scientific computing
    Pandas GUIPandas GUI
    NuriStatFree open-source SPSS alternative — menu-driven desktop statistics (t-tests, ANOVA, regression, survival analysis, ROC) with SPSS .sav import/export
    PolarsFast DataFrame library for Rust and Python, designed as a faster alternative to Pandas
    CiteMefree academic citation generator with a built-in reference checker that flags fabricated or hallucinated references. Searches 11+ scholarly databases (OpenAlex, PubMed, Semantic Scholar, CrossRef, SciELO), formats 40+ citation styles, and offers a public API. No sign-up; available in English, Spanish, Portuguese, French, and German.
    Scikit-LearnMachine Learning in Python
    NumPyNumPy is fundamental for scientific computing with Python. It supports large, multi-dimensional arrays and matrices and includes an assortment of high-level mathematical functions to operate on these arrays.
    VaexVaex is a Python library that allows you to visualize large datasets and calculate statistics at high speeds.
    SciPySciPy works with NumPy arrays and provides efficient routines for numerical integration and optimization.
    Data Science ToolboxCoursera Course
    Data Science ToolboxBlog
    Wolfram Data Science PlatformTake numerical, textual, image, GIS or other data and give it the Wolfram treatment, carrying out a full spectrum of data science analysis and visualization and automatically generate rich interactive reports—all powered by the revolutionary knowledge-based Wolfram Language.
    DatadogSolutions, code, and devops for high-scale data science.
    VarianceBuild powerful data visualizations for the web without writing JavaScript
    Kite Development KitThe Kite Software Development Kit (Apache License, Version 2.0), or Kite for short, is a set of libraries, tools, examples, and documentation focused on making it easier to build systems on top of the Hadoop ecosystem.
    Domino Data LabsRun, scale, share, and deploy your models — without any infrastructure or setup.
    Apache FlinkA platform for efficient, distributed, general-purpose data processing.
    Apache HamaApache Hama is an Apache Top-Level open source project, allowing you to do advanced analytics beyond MapReduce.
    WekaWeka is a collection of machine learning algorithms for data mining tasks.
    OctaveGNU Octave is a high-level interpreted language, primarily intended for numerical computations.(Free Matlab)
    Apache SparkLightning-fast cluster computing
    Hydrosphere Mista service for exposing Apache Spark analytics jobs and machine learning models as realtime, batch or reactive web services.
    Data MechanicsA data science and engineering platform making Apache Spark more developer-friendly and cost-effective.
    CaffeDeep Learning Framework
    TorchA SCIENTIFIC COMPUTING FRAMEWORK FOR LUAJIT
    Nervana’s python based Deep Learning FrameworkIntel® Nervana™ reference deep learning framework committed to best performance on all hardware.
    SkaleHigh performance distributed data processing in NodeJS
    AerosolveA machine learning package built for humans.
    Intel frameworkIntel® Deep Learning Framework
    DatawrapperAn open source data visualization platform helping everyone to create simple, correct and embeddable charts. Also at github.com
    Tensor FlowTensorFlow is an Open Source Software Library for Machine Intelligence
    Natural Language ToolkitAn introductory yet powerful toolkit for natural language processing and classification
    FunASRIndustrial-grade speech recognition toolkit supporting 50+ languages with built-in VAD, punctuation, speaker diarization, and emotion detection. OpenAI-compatible API server included.
    Annotation LabFree End-to-End No-Code platform for text annotation and DL model training/tuning. Out-of-the-box support for Named Entity Recognition, Classification, Relation extraction and Assertion Status Spark NLP models. Unlimited support for users, teams, projects, documents.
    nlp-toolkit for node.jsThis module covers some basic nlp principles and implementations. The main focus is performance. When we deal with sample or training data in nlp, we quickly run out of memory. Therefore every implementation in this module is written as stream to only hold that data in memory that is currently processed at any step.
    Juliahigh-level, high-performance dynamic programming language for technical computing
    IJuliaa Julia-language backend combined with the Jupyter interactive environment
    Apache ZeppelinWeb-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more
    FeaturetoolsAn open source framework for automated feature engineering written in python
    OptimusCleansing, pre-processing, feature engineering, exploratory data analysis and easy ML with PySpark backend.
    AlbumentationsА fast and framework agnostic image augmentation library that implements a diverse set of augmentation techniques. Supports classification, segmentation, and detection out of the box. Was used to win a number of Deep Learning competitions at Kaggle, Topcoder and those that were a part of the CVPR workshops.
    DVCAn open-source data science version control system. It helps track, organize and make data science projects reproducible. In its very basic scenario it helps version control and share large data and model files.
    Lambdois a workflow engine that significantly simplifies data analysis by combining in one analysis pipeline (i) feature engineering and machine learning (ii) model training and prediction (iii) table population and column evaluation.
    FeastA feature store for the management, discovery, and access of machine learning features. Feast provides a consistent view of feature data for both model training and model serving.
    PolyaxonA platform for reproducible and scalable machine learning and deep learning.
    UBIAIEasy-to-use text annotation tool for teams with most comprehensive auto-annotation features. Supports NER, relations and document classification as well as OCR annotation for invoice labeling
    TrainsAuto-Magical Experiment Manager, Version Control & DevOps for AI
    HopsworksOpen-source data-intensive machine learning platform with a feature store. Ingest and manage features for both online (MySQL Cluster) and offline (Apache Hive) access, train and serve models at scale.
    MindsDBMindsDB is an Explainable AutoML framework for developers. With MindsDB you can build, train and use state of the art ML models in as simple as one line of code.
    LightwoodA Pytorch based framework that breaks down machine learning problems into smaller blocks that can be glued together seamlessly with an objective to build predictive models with one line of code.
    AWS Data WranglerAn open-source Python package that extends the power of Pandas library to AWS connecting DataFrames and AWS data related services (Amazon Redshift, AWS Glue, Amazon Athena, Amazon EMR, etc).
    Amazon RekognitionAWS Rekognition is a service that lets developers working with Amazon Web Services add image analysis to their applications. Catalog assets, automate workflows, and extract meaning from your media and applications.
    Amazon TextractAutomatically extract printed text, handwriting, and data from any document.
    Amazon Lookout for VisionSpot product defects using computer vision to automate quality inspection. Identify missing product components, vehicle and structure damage, and irregularities for comprehensive quality control.
    Amazon CodeGuruAutomate code reviews and optimize application performance with ML-powered recommendations.
    CMLAn open source toolkit for using continuous integration in data science projects. Automatically train and test models in production-like environments with GitHub Actions & GitLab CI, and autogenerate visual reports on pull/merge requests.
    DaskAn open source Python library to painlessly transition your analytics code to distributed computing systems (Big Data)
    DuckDBAn in-process SQL OLAP database management system
    StatsmodelsA Python-based inferential statistics, hypothesis testing and regression framework
    GensimAn open-source library for topic modeling of natural language text
    spaCyA performant natural language processing toolkit
    Grid StudioGrid studio is a web-based spreadsheet application with full integration of the Python programming language.
    Python Data Science HandbookPython Data Science Handbook: full text in Jupyter Notebooks
    ShapleyA data-driven framework to quantify the value of classifiers in a machine learning ensemble.
    DAGsHubA platform built on open source tools for data, model and pipeline management.
    DeepnoteA new kind of data science notebook. Jupyter-compatible, with real-time collaboration and running in the cloud.
    ValohaiAn MLOps platform that handles machine orchestration, automatic reproducibility and deployment.
    PyMC3A Python Library for Probabalistic Programming (Bayesian Inference and Machine Learning)
    PyStanPython interface to Stan (Bayesian inference and modeling)
    hmmlearnUnsupervised learning and inference of Hidden Markov Models
    Chaos GeniusML powered analytics engine for outlier/anomaly detection and root cause analysis
    NimbleboxA full-stack MLOps platform designed to help data scientists and machine learning practitioners around the world discover, create, and launch multi-cloud apps from their web browser.
    TowheeA Python library that helps you encode your unstructured data into embeddings.
    LineaPyEver been frustrated with cleaning up long, messy Jupyter notebooks? With LineaPy, an open source Python library, it takes as little as two lines of code to transform messy development code into production pipelines.
    envd🏕️ machine learning development environment for data science and AI/ML engineering teams
    Explore Data Science LibrariesA search engine 🔎 tool to discover & find a curated list of popular & new libraries, top authors, trending project kits, discussions, tutorials & learning resources
    MLEM🐶 Version and deploy your ML models following GitOps principles
    MLflowMLOps framework for managing ML models across their full lifecycle
    cleanlabPython library for data-centric AI and automatically detecting various issues in ML datasets
    AutoGluonAutoML to easily produce accurate predictions for image, text, tabular, time-series, and multi-modal data
    Arize AIArize AI community tier observability tool for monitoring machine learning models in production and root-causing issues such as data quality and performance drift.
    Aureo.ioAureo.io is a low-code platform that focuses on building artificial intelligence. It provides users with the capability to create pipelines, automations and integrate them with artificial intelligence models – all with their basic data.
    ERD LabFree cloud based entity relationship diagram (ERD) tool made for developers.
    Arize-PhoenixMLOps in a notebook - uncover insights, surface problems, monitor, and fine tune your models.
    CometAn MLOps platform with experiment tracking, model production management, a model registry, and full data lineage to support your ML workflow from training straight through to production.
    OpikEvaluate, test, and ship LLM applications across your dev and production lifecycles.
    SynthicalAI-powered collaborative environment for research. Find relevant papers, create collections to manage bibliography, and summarize content — all in one place
    teeplotWorkflow tool to automatically organize data visualization output
    StreamlitApp framework for Machine Learning and Data Science projects
    GradioCreate customizable UI components around machine learning models
    Weights & BiasesExperiment tracking, dataset versioning, and model management
    DVCOpen-source version control system for machine learning projects
    OptunaAutomatic hyperparameter optimization software framework
    Ray TuneScalable hyperparameter tuning library
    Apache AirflowPlatform to programmatically author, schedule, and monitor workflows
    PrefectWorkflow management system for modern data stacks
    KedroOpen-source Python framework for creating reproducible, maintainable data science code
    HamiltonLightweight library to author and manage reliable data transformations
    SHAPGame theoretic approach to explain the output of any machine learning model
    InterpretMLInterpretML implements the Explainable Boosting Machine (EBM), a modern, fully interpretable machine learning model based on Generalized Additive Models (GAMs). This open-source package also provides visualization tools for EBMs, other glass-box models, and black-box explanations
    LIMEExplaining the predictions of any machine learning classifier
    flyteWorkflow automation platform for machine learning
    dbtData build tool
    zasperSupercharged IDE for Data Science
    skrubA Python library to ease preprocessing and feature engineering for tabular machine learning
    CodeflashShip Blazing-Fast Python Code — Every Time
    Hugging FacePopular open platform for sharing ML models, datasets, and collaborating on NLP and generative AI projects.
    Chinese-EliteAn open-source project that automatically maps relationship networks by parsing public data using LLMs and visualizes it as an interactive graph.
    DesbordanteAn open-source data profiler specifically focused on discovery and validation of complex patterns, such as numerical association rules, differential dependencies, denial constraints, and more.
    dna-claude-analysisPersonal genome analysis toolkit with Python scripts analyzing raw DNA data across 17 categories (health risks, ancestry, pharmacogenomics, nutrition, psychology, and more) and generating a terminal-style single-page HTML visualization.
    RunMatFast MATLAB-syntax runtime with automatic CPU/GPU execution and fused array kernels.
    TurbostreamA terminal UI for experimenting with custom rule engines and selective LLM analysis on real-time data streams, without worrying about streaming infra or backpressure.
    WFGY ProblemMapOpen source “failure atlas” of 16 recurring issues in LLM and RAG pipelines, with observable symptoms and suggested fixes for data science teams.
    DeploybaseTrack real-time GPU and LLM pricing across all cloud and inference providers.
    DeepAnalyzeAn agentic LLM for autonomous data science, which can autonomously complete a wide range of data science tasks without human intervention.
    DiscoSuperhuman exploratory data analysis. Finds the feature interactions and subgroup effects in tabular data that LLMs and manual exploration miss — with p-values, effect sizes, and literature citations. Free for public data.
    AI for DatabaseChat with your database in natural language — no SQL needed. Get instant insights, build self-refreshing dashboards, and trigger automated workflows based on database changes.
    Crypto Pump ScannerAI-powered cryptocurrency trading bot with LSTM neural network (84.6% accuracy). Real-time pump detection, walk-forward validated models, multi-exchange support (Bybit, Binance, OKX, Gate.io). Open source.
    Future AGIOpen-source platform to simulate, evaluate, trace, guardrail, route, and optimize LLM and AI agent apps in one feedback loop, so agents don’t just get monitored, they self-improve. Self-hostable. Apache-2.0.

    Literature and Media

    ^ back to top ^

    このセクションには、追加の読み物、視聴できるチャンネル、および聴ける講演が含まれています。

    Books

    ^ back to top ^

    Book Deals (Affiliated)

    Journals, Publications and Magazines

    ^ back to top ^

    Newsletters

    ^ back to top ^

    • AI Weekly - 業界リーダーによるAIインテリジェンスのキュレーションされた要約。2017年から3回/週、40,000人以上のサブスクリプションを持つ。
    • DataTalks.Club. データに関連する話題についての週刊ニュースレター。 Archive.
    • The Analytics Engineering Roundup. データサイエンスに関するニュースレター。 Archive.
    • Techpresso. AI、機械学習、テクノロジー分野における最も影響力のある進展をカバーする無料の日刊ニュースレター。 Archive.
    • DiamantAI. 実用的なAIエンジニアリングと生成AIをシンプルに説明：RAG、エージェント、LLMの応用パターンについての開発者向け解説。
    • Bamboo Weekly - 現実の出来事や公開データに基づいた週次pandas練習問題。完全な解説付き。2年以上前の問題は無料であり、現在の号の最初の2問と解答も無料。 Archive.

    Mailing lists

    ^ back to top ^

    Bloggers

    ^ back to top ^

    Presentations

    ^ back to top ^

    Podcasts

    ^ back to top ^

    YouTube Videos & Channels

    ^ back to top ^

    Socialize

    ^ back to top ^

    以下にいくつかのソーシャルメディアのリンクがあります。他のデータサイエンティストとつながりましょう！

    Facebook Accounts

    ^ back to top ^

    Twitter Accounts

    ^ back to top ^

    TwitterDescription
    Big Data CombineRapid-fire, live tryouts for data scientists seeking to monetize their models as trading strategies
    Big Data ManiaData Viz Wiz, Data Journalist, Growth Hacker, Author of Data Science for Dummies (2015)
    Big Data ScienceBig Data, Data Science, Predictive Modeling, Business Analytics, Hadoop, Decision and Operations Research.
    Charlie GreenbackerDirector of Data Science at @ExploreAltamira
    Chris SaidData scientist at Twitter
    Clare CorthellDev, Design, Data Science @mattermark #hackerei
    DADI Charles-Abner#datascientist @Ekimetrics. , #machinelearning #dataviz #DynamicCharts #Hadoop #R #Python #NLP #Bitcoin #dataenthousiast
    Data Science CentralData Science Central is the industry’s single resource for Big Data practitioners.
    Data Science LondonData Science. Big Data. Data Hacks. Data Junkies. Data Startups. Open Data
    Data Science ReneeDocumenting my path from SQL Data Analyst pursuing an Engineering Master’s Degree to Data Scientist
    Data Science ReportMission is to help guide & advance careers in Data Science & Analytics
    Data Science TipsTips and Tricks for Data Scientists around the world! #datascience #bigdata
    Data VizzardDataViz, Security, Military
    DataScienceX
    deeplearning4j
    DJ PatilWhite House Data Chief, VP @ RelateIQ.
    Domino Data Lab
    Drew ConwayData nerd, hacker, student of conflict.
    Emilio Ferrara#Networks, #MachineLearning and #DataScience. I work on #Social Media. Postdoc at @IndianaUniv
    Erin BartoloRunning with #BigData—enjoying a love/hate relationship with its hype. @iSchoolSU #DataScience Program Mgr.
    Greg RedaWorking @ GrubHub about data and pandas
    Gregory PiatetskyKDnuggets President, Analytics/Big Data/Data Mining/Data Science expert, KDD & SIGKDD co-founder, was Chief Scientist at 2 startups, part-time philosopher.
    Hadley WickhamChief Scientist at RStudio, and an Adjunct Professor of Statistics at the University of Auckland, Stanford University, and Rice University.
    Hakan KardasData Scientist
    Hilary MasonData Scientist in Residence at @accel.
    Jeff HammerbacherReTweeting about data science
    John Myles WhiteScientist at Facebook and Julia developer. Author of Machine Learning for Hackers and Bandit Algorithms for Website Optimization. Tweets reflect my views only.
    Juan Miguel LavistaPrincipal Data Scientist @ Microsoft Data Science Team
    Julia EvansHacker - Pandas - Data Analyze
    Kenneth CukierThe Economist’s Data Editor and co-author of Big Data (https://www.big-data-book.com/).
    Kevin DavenportOrganizer of https://www.meetup.com/San-Diego-Data-Science-R-Users-Group/
    Kevin MarkhamData science instructor, and founder of Data School
    Kim ReesInteractive data visualization and tools. Data flaneur.
    Kirk BorneDataScientist, PhD Astrophysicist, Top #BigData Influencer.
    Linda RegberData storyteller, visualizations.
    Luis ReiPhD Student. Programming, Mobile, Web. Artificial Intelligence, Intelligent Robotics Machine Learning, Data Mining, Natural Language Processing, Data Science.
    Mark StevensonData Analytics Recruitment Specialist at Salt (@SaltJobs) Analytics - Insight - Big Data - Data science
    Matt HarrisonOpinions of full-stack Python guy, author, instructor, currently playing Data Scientist. Occasional fathering, husbanding, organic gardening.
    Matthew RussellMining the Social Web.
    Mert NuhoğluData Scientist at BizQualify, Developer
    Monica RogatiData @ Jawbone. Turned data into stories & products at LinkedIn. Text mining, applied machine learning, recommender systems. Ex-gamer, ex-machine coder; namer.
    Noah IliinskyVisualization & interaction designer. Practical cyclist. Author of vis books: https://www.oreilly.com/pub/au/4419
    Paul MillerCloud Computing/ Big Data/ Open Data Analyst & Consultant. Writer, Speaker & Moderator. Gigaom Research Analyst.
    Peter SkomorochCreating intelligent systems to automate tasks & improve decisions. Entrepreneur, ex-Principal Data Scientist @LinkedIn. Machine Learning, ProductRei, Networks
    Prash ChanSolution Architect @ IBM, Master Data Management, Data Quality & Data Governance Blogger. Data Science, Hadoop, Big Data & Cloud.
    Quora Data ScienceQuora’s data science topic
    R-BloggersTweet blog posts from the R blogosphere, data science conferences, and (!) open jobs for data scientists.
    Rand Hindi
    Randy OlsonComputer scientist researching artificial intelligence. Data tinkerer. Community leader for @DataIsBeautiful. #OpenScience advocate.
    Recep ErolData Science geek @ UALR
    Ryan OrbanData scientist, genetic origamist, hardware aficionado
    Sean J. TaylorSocial Scientist. Hacker. Facebook Data Science Team. Keywords: Experiments, Causal Inference, Statistics, Machine Learning, Economics.
    Silvia K. Spiva#DataScience at Cisco
    Harsh B. GuptaData Scientist at BBVA Compass
    Spencer NelsonData nerd
    Talha OzEnjoys ABM, SNA, DM, ML, NLP, HI, Python, Java. Top percentile Kaggler/data scientist
    Tasos SkarlatidisComplex Event Processing, Big Data, Artificial Intelligence and Machine Learning. Passionate about programming and open-source.
    Terry TimkoInfoGov; Bigdata; Data as a Service; Data Science; Open, Social & Business Data Convergence
    Tony BaerIT analyst with Ovum covering Big Data & data management with some systems engineering thrown in.
    Tony OjedaData Scientist , Author , Entrepreneur. Co-founder @DataCommunityDC. Founder @DistrictDataLab. #DataScience #BigData #DataDC
    Vamshi AmbatiData Science @ PayPal. #NLP, #machinelearning; PhD, Carnegie Mellon alumni (Blog: https://allthingsds.wordpress.com )
    Wes McKinneyPandas (Python Data Analysis library).
    WileyEdSenior Manager - @Seagate Big Data Analytics @McKinsey Alum #BigData + #Analytics Evangelist #Hadoop, #Cloud, #Digital, & #R Enthusiast
    WNYC Data News TeamThe data news crew at @WNYC. Practicing data-driven journalism, making it visual, and showing our work.
    Alexey GrigorevData science author
    İlker ArslanData science author. Shares mostly about Julia programming
    INEVITABLEAI & Data Science Start-up Company based in England, UK
    Jan Oliver RüdigerML, DL and Data Science - with a focus on text-/data-mining

    Telegram Channels

    ^ back to top ^

    • Open Data Science – 最初のTelegramデータサイエンスチャンネル。AI、ビッグデータ、機械学習、統計、一般数学およびそれらの応用に関するすべての技術的および人気のトピックをカバーしています
    • Loss function porn — DS/MLテーマに関する美しい投稿。動画やグラフィックによる可視化を含む。
    • Machinelearning – 毎日のMLニュース

    Slack Communities

    top

    GitHub Groups

    Data Science Competitions

    いくつかのデータマイニングコンペティションプラットフォーム

    Fun

    Infographics

    ^ back to top ^

    PreviewDescription
    Key differences of a data scientist vs. data engineer
    A visual guide to Becoming a Data Scientist in 8 Steps by DataCamp (img)
    Mindmap on required skills (img)
    Swami Chandrasekaran made a Curriculum via Metro map.
    by @kzawadz via twitter
    By Data Science Central
    Data Science Wars: R vs Python
    How to select statistical or machine learning techniques
    Choosing the Right Estimator
    The Data Science Industry: Who Does What
    Data Science Venn Euler Diagram
    Different Data Science Skills and Roles from Springboard
    Data Fallacies To AvoidA simple and friendly way of teaching your non-data scientist/non-statistician colleagues how to avoid mistakes with data. From Geckoboard’s Data Literacy Lessons.

    Datasets

    ^ back to top ^

    Comics

    ^ back to top ^

    Other Awesome Lists

    Hobby