【工作流工具】精选计算数据分析工作流系统
Chinese, Simplified
Computational Data Analysis Workflow Systems
Permalink: https://s.apache.org/existing-workflow-systems
An incomplete list
Please add new entries at the bottom.
See also: https://github.com/pditommaso/awesome-pipeline
- Arvados http://arvados.org
- Taverna http://www.taverna.org.uk/
- Galaxy http://galaxyproject.org/
- SHIWA https://www.shiwa-workflow.eu/
- Oozie https://oozie.apache.org/
- DNANexus https://wiki.dnanexus.com/API-Specification-v1.0.0/IO-and-Run-Specifications# https://wiki.dnanexus.com/API-Specification-v1.0.0/Workflows-and-Analyses#
- BioDT http://www.biodatomics.com/
- Agave http://agaveapi.co/live-docs/
- DiscoveryEnvironment http://www.iplantcollaborative.org/ci/discovery-environment
- Wings http://www.wings-workflows.org/
- Knime https://www.knime.org/
- make, rake, drake, ant, scons & many others. Software development relies heavily on tools to manage workflows related to compiling and packaging applications. For the most part these are file based and usually run on a single node, usually supporting parallel steps (make -j) and in some cases able to dispatch build steps to other machines (https://code.google.com/p/distcc/) https://github.com/Factual/drake
- Snakemake https://bitbucket.org/snakemake/snakemake
- BPipe http://bpipe.org
- Ruffus https://code.google.com/p/ruffus/
- NextFlow http://nextflow.io
- Luigi http://github.com/spotify/luigi https://luigi.readthedocs.io
- SciLuigi. Helper library built on top of Luigi to ease development of Scientific workflows in Luigi: http://github.com/pharmbio/sciluigi
- Luigi Analysis Workflow (LAW) https://github.com/riga/law
- GATK Queue https://www.broadinstitute.org/gatk/guide/topic?name=queue
- Yabi https://ccg.murdoch.edu.au/yabi
- seqware Workflows are written in Java and executed using the Oozie Workflow Engine on Hadoop or SGE clusters. Uses Zip64 files to group the workflow definition file, workflow itself, sample settings, and data dependencies in a single file that can be exchanged between SeqWare users or archived. https://seqware.github.io/ https://seqware.github.io/docs/6-pipeline/
- Ketrew https://github.com/hammerlab/ketrew
- Pegasus http://pegasus.isi.edu/
- Apache Airflow https://github.com/apache/airflow
- Cosmos https://cosmos.hms.harvard.edu/documentation/index.html http://bioinformatics.oxfordjournals.org/content/early/2014/07/24/bioinformatics.btu385.full [paper] Cosmos2: https://github.com/LPM-HMS/COSMOS2 http://cosmos.hms.harvard.edu/COSMOS2/
- Pinball https://github.com/pinterest/pinball
- bcbio https://bcbio-nextgen.readthedocs.org/en/latest/
- Chronos https://github.com/mesos/chronos
- Azkaban https://azkaban.github.io/
- Apache NiFi https://nifi.apache.org/docs/nifi-docs/html/overview.html
- flowr (R-based) http://docs.flowr.space/ https://github.com/sahilseth/flowr
- Mistral https://github.com/arteria-project https://wiki.openstack.org/wiki/Mistral#What_is_Mistral.3F https://docs.openstack.org/mistral/latest/user/wf_lang_v2.html
- nipype http://nipy.org/nipype/
- End of Day https://github.com/joestubbs/endofday
- BioDSL https://github.com/maasha/BioDSL
- BigDataScript http://pcingola.github.io/BigDataScript/
- Omics Pipe: uses Ruffus http://sulab.scripps.edu/omicspipe/
- Ensembl Hive https://github.com/Ensembl/ensembl-hive
- QuickNGS http://bifacility.uni-koeln.de/quickngs/web
- GenePattern http://www.broadinstitute.org/cancer/software/genepattern/
- Chipster http://chipster.csc.fi/
- The Genome Modeling System https://github.com/genome/gms
- Cuneiform, A Functional Workflow Language https://github.com/joergen7/cuneiform http://www.cuneiform-lang.org/
- Anvaya http://www.ncbi.nlm.nih.gov/pubmed/22809419 http://webapp.cabgrid.res.in/biocomp/Anvaya/ANVAYA_Main.html#HOWTO_INSTALL_ANVAYA
- Makeflow http://ccl.cse.nd.edu/software/makeflow/
- Airavata http://airavata.apache.org/
- Pyflow https://github.com/Illumina/pyflow
- Cluster Flow http://clusterflow.io
- Unipro UGENE http://ugene.net/ https://dx.doi.org/10.7717/peerj.644
- CloudSlang http://www.cloudslang.io/
- Stacks http://catchenlab.life.illinois.edu/stacks/
- Leaf http://www.francesconapolitano.it/leaf/index.html
- omictools http://omictools.com/
- Job Description Language. The Job Description Language, JDL, is a high-level, user-oriented language based on Condor classified advertisements for describing jobs and aggregates of jobs such as Direct Acyclic Graphs and Collections. https://edms.cern.ch/ui/file/590869/1/WMS-JDL.pdf
- YAWL yet another workflow language http://dx.doi.org/10.1016/j.is.2004.02.002 http://www.yawlfoundation.org/
- Triquetrum https://projects.eclipse.org/projects/technology.triquetrum https://github.com/eclipse/triquetrum/
- Kronos https://github.com/jtaghiyar/kronos
- qsubsec http://doi.org/10.1093/bioinformatics/btv698 https://github.com/alastair-droop/qsubsec
- YesWorkflow http://yesworkflow.org
- gwf - Grid WorkFlow https://github.com/gwforg/gwf http://gwf.readthedocs.io/
- Fireworks. https://pythonhosted.org/FireWorks/
- NGLess: NGS with less work http://ngless.rtfd.io
- pypipegraph https://github.com/TyberiusPrime/pypipegraph
- Cromwell https://github.com/broadinstitute/cromwell
- Dagobah - Simple DAG-based job scheduler in Python. https://github.com/thieman/dagobah
- sushi https://github.com/uzh/sushi
- Clinical Trial Processor - A program for processing clinical trials data. http://mircwiki.rsna.org/index.php?title=MIRC_CTP
- Noodles http://nlesc.github.io/noodles/
- Swift & Swift/T http://swift-lang.org/main/ http://swift-lang.org/Swift-T/
- Consonance (runs SeqWare & CWL) https://github.com/Consonance/consonance/wiki
- Dog https://github.com/dogtools/dog
- Produce https://github.com/texttheater/produce
- LONI Pipeline http://pipeline.loni.usc.edu/
- Cpipe https://github.com/MelbourneGenomics/cpipe
- AWE https://github.com/MG-RAST/AWE
- (Py)COMPSs https://www.bsc.es/research-and-development/software-and-apps/software-list/comp-superscalar/
- KLIKO https://github.com/gijzelaerr/kliko
- SoS Workflow https://github.com/vatlab/SoS https://vatlab.github.io/sos-docs/ https://doi.org/10.1093/bioinformatics/bty405 https://doi.org/10.1371/journal.pcbi.1006843
- XNAT Pipeline Engine https://wiki.xnat.org/display/XNAT/Pipeline+Engine https://wiki.xnat.org/display/XNAT/XNAT+Pipeline+Development+Schema
- Metapipe https://github.com/TorkamaniLab/metapipe
- OCCAM (Open Curation for Computer Architecture Modeling) https://occam.cs.pitt.edu/
- Copernicus http://www.copernicus-computing.org
- iRODS Rule Language https://github.com/samuell/irods-cheatsheets/blob/master/irods-rule-lang-full-guide.md
- VisTrails https://www.vistrails.org
- Bionode Watermill https://github.com/bionode/bionode-watermill
- BIOVIA Pipeline Pilot Overview http://accelrys.com/products/collaborative-science/biovia-pipeline-pilot/
- Dagman A meta-scheduler for HTCondor https://research.cs.wisc.edu/htcondor/dagman/dagman.html
- UNICORE https://www.unicore.eu/docstore/workflow-7.6.0/workflow-manual.html#wf_dialect
- Toil (A scalable, efficient, cross-platform and easy-to-use workflow engine in pure Python) https://github.com/BD2KGenomics/toil
- Cylc (a workflow engine for cycling systems) https://cylc.github.io/
- Autodesk Cloud Compute Canon https://github.com/Autodesk/cloud-compute-cannon
- Civet https://github.com/TheJacksonLaboratory/civet
- Cumulus https://github.com/Kitware/cumulus
- High-performance integrated virtual environment (HIVE) https://hive.biochemistry.gwu.edu
- Cloudgene http://cloudgene.uibk.ac.at/cloudgene-yaml
- FASTR https://bitbucket.org/bigr_erasmusmc/fastr/ http://fastr.readthedocs.io/en/stable/
- BioMake https://github.com/evoldoers/biomake http://dx.doi.org/10.1101/093245
- remake https://github.com/richfitz/remake
- SciFloware http://www-sop.inria.fr/members/Didier.Parigot/pmwiki/Scifloware/
- OpenAlea http://openalea.gforge.inria.fr/dokuwiki/doku.php https://hal.archives-ouvertes.fr/hal-01166298/file/openalea-PradalCohen-Boulakia.pdf
- COMBUSTI/O https://github.com/jarlebass/combustio http://hdl.handle.net/10037/9361
- BioCloud https://github.com/ccwang002/biocloud-server-kai http://doi.org/10.6342/NTU201601295
Triana http://www.trianacode.org/(website seems to have been taken over by SEO)- Kepler https://kepler-project.org/
- Anduril http://anduril.org/site/
- dgsh http://www.dmst.aueb.gr/dds/sw/dgsh/
- EDGE bioinformatics: Empowering the Development of Genomics Expertise https://bioedge.lanl.gov/edge_ui/ http://edge.readthedocs.io/ https://lanl-bioinformatics.github.io/EDGE/
- Pachyderm http://pachyderm.io/ http://pachyderm.readthedocs.io/en/stable/advanced/advanced.html
- Digdag https://www.digdag.io/
- Agua / Automated Genomics Utilities Agent http://aguadev.org
- BioDepot Workflow Builder (BwB) https://github.com/BioDepot/BioDepot-workflow-builder https://doi.org/10.1101/099010
- IMP: a pipeline for reproducible reference-independent integrated metagenomic and metatranscriptomic analyses http://r3lab.uni.lu/web/imp/ https://doi.org/10.1186/s13059-016-1116-8
- Butler https://github.com/llevar/butler
- adage / yadage https://github.com/diana-hep/adage https://github.com/diana-hep/yadage
- HI-WAY: Execution of Scientific Workflows on Hadoop YARN https://github.com/marcbux/Hi-WAY https://openproceedings.org/2017/conf/edbt/paper-248.pdf
- OpenMOLE https://github.com/openmole/openmole https://www.openmole.org/ https://doi.org/10.3389/fninf.2017.00021
- Biopet https://github.com/biopet/biopet
- Nephele https://nephele.niaid.nih.gov/
- TOPPAS http://doi.org/10.1021/pr300187f
- SBpipe https://pdp10.github.io/sbpipe/ https://github.com/pdp10/sbpipe https://doi.org/10.1186/s12918-017-0423-3
- Dray http://dray.it/
- GenomeVIP https://github.com/ding-lab/GenomeVIP https://doi.org/10.1101/gr.211656.116
- GridSAM https://sourceforge.net/projects/gridsam/
- Roddy https://github.com/eilslabs/Roddy
- SciFlo (historical; doesn't seem to be maintained anymore) https://web.archive.org/web/20161118011409/https://sciflo.jpl.nasa.gov/SciFloWiki/FrontPage
- GNU Guix Workflow Language https://git.roelj.com/guix/gwl.git#gnu-guix-workflow-language-extension https://github.com/UMCUGenetics/guix-workflows/blob/master/umcu/workflows/rnaseq.scm
- Porcupine https://timvanmourik.github.io/Porcupine/
- Parsl (a Parallel Scripting Library for Python) http://parsl-project.org
- ECFLOW (Workflow primarily for Meteorological Applications) https://software.ecmwf.int/wiki/display/ECFLOW/ecflow+home
- Ophidia http://ophidia.cmcc.it/
- WebLicht https://weblicht.sfs.uni-tuebingen.de/
- GATE Cloud https://cloud.gate.ac.uk/
- SCIPION http://scipion.cnb.csic.es/m/home/ https://github.com/I2PC/scipion/wiki/Creating-a-Protocol
- Ergatis http://ergatis.sourceforge.net/
- TIGR "Workflow" https://sourceforge.net/projects/tigr-workflow/ http://tigr-workflow.sourceforge.net/
- Archivematica https://wiki.archivematica.org/Main_Page (A preservation workflow system that implements the ISO-OAIS standard using gearman/MCP)
- Martian http://martian-lang.org/about/
- BioMAJ http://genouest.github.io/biomaj/
- Conveyor http://conveyor.cebitec.uni-bielefeld.de (retired). https://academic.oup.com/bioinformatics/article/27/7/903/230562/Conveyor-a-workflow-engine-for-bioinformatic
- Biopipe http://www.biopipe.org (appears to be defunct) https://www.ncbi.nlm.nih.gov/pmc/articles/PMC403782/
- Wildfire http://wildfire.bii.a-star.edu.sg/ https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-6-69
- BioWBI http://bioinformatics.hsanmartino.it/bits_library/library/00079.pdf
- BioWMS http://bioinformatics.hsanmartino.it/bits_library/library/00568.pdf
- BioMoby http://biomoby.open-bio.org/ https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-7-523
- SIBIOS http://ieeexplore.ieee.org/document/1309094/
- NGSANE https://github.com/BauerLab/ngsane https://academic.oup.com/bioinformatics/article/30/10/1471/266879/NGSANE-a-lightweight-production-informatics
- Pwrake https://github.com/misshie/Workflows https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3180464/
- Nesoni https://github.com/Victorian-Bioinformatics-Consortium/nesoni
- Skam http://skam.sourceforge.net/skam-intro.html
- TREVA http://bioinformatics.petermac.org/treva/ http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0095217
- EGene https://www.semanticscholar.org/paper/EGene-a-configurable-pipeline-generation-system-fo-Durham-Kashiwabara/4c0656195b5efcdd3aa7bdcb55fc95a957c150aa https://academic.oup.com/bioinformatics/article/30/18/2659/2475637/EuGene-PP-a-next-generation-automated-annotation
- WEP https://bioinformatics.cineca.it/wep/ https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-14-S7-S11
- Microbase http://www.microbasecloud.com/
- e-Science Central http://www.esciencecentral.co.uk/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3538293/
- Cyrille2 https://bmcbioinformatics.biomedcentral.com/articles/10.1186/1471-2105-9-96
- PaPy https://code.google.com/archive/p/papy/ https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3051902/
- JobCenter https://github.com/yeastrc/jobcenter https://scfbm.biomedcentral.com/articles/10.1186/1751-0473-7-8
- CoreFlow https://www.ncbi.nlm.nih.gov/pubmed/24503186
- dynamic-pipeline https://code.google.com/archive/p/dynamic-pipeline/
- XiP http://xip.hgc.jp/wiki/en/Main_Page https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3530915/
- Eoulsan http://www.outils.genomique.biologie.ens.fr/eoulsan/ https://www.ncbi.nlm.nih.gov/pubmed/22492314
- CloudDOE http://clouddoe.iis.sinica.edu.tw/
- BioPig https://github.com/JGI-Bioinformatics/biopig https://www.ncbi.nlm.nih.gov/pubmed/24021384
- SeqPig https://github.com/HadoopGenomics/SeqPig https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3866557/
- zymake http://www-personal.umich.edu/~ebreck/code/zymake/
- JMS https://github.com/RUBi-ZA/JMS http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0134273
- CLC Genomics Workbench https://www.qiagenbioinformatics.com/products/clc-genomics-workbench/
- NG6 http://ng6.toulouse.inra.fr/ https://www.ncbi.nlm.nih.gov/pubmed/22958229
- VIBE http://www.incogen.com/vibe/
- WDL (Workflow Description Language) https://github.com/broadinstitute/wdl
- SciFlow https://github.com/kaizhang/SciFlow (not to be confused with SciFloware and SciFlo).
- Bioshake https://github.com/PapenfussLab/bioshake
- SciPipe http://scipipe.org
- Kapacitor / TICKscripts https://docs.influxdata.com/kapacitor/v1.3//tick/
- AiiDA: Automated Interactive Infrastructure and Database for Computational Science http://www.aiida.net/
- Reflow: a language and runtime for distributed, integrated data processing in the cloud https://github.com/grailbio/reflow
- Resolwe: an open source dataflow package for Django framework https://github.com/genialis/resolwe
- Yahoo! Pipes (historical) https://en.wikipedia.org/wiki/Yahoo!_Pipes
- Walrus https://github.com/fjukstad/walrus
- Apache Beam https://beam.apache.org/
- CLOSHA https://closha.kobic.re.kr/ https://www.bioexpress.re.kr/go_tutorial http://docplayer.net/19700397-Closha-manual-ver1-1-kobic-korean-bioinformation-center-kogun82-kribb-re-kr-2016-05-08-bioinformatics-workflow-management-system-in-bio-express.html https://doi.org/10.1186/s12859-018-2019-3
- WopMars https://github.com/aitgon/wopmars http://wopmars.readthedocs.io/
- flowing-clj https://github.com/stain/flowing-clj
- Plumbing and Graph https://github.com/plumatic/plumbing
- LabView http://www.ni.com/en-us/shop/labview.html
- MyOpenLab http://myopenlab.org/
- Max/MSP https://cycling74.com/products/max/
- NoFlo https://noflojs.org/
- Flowstone http://www.dsprobotics.com/flowstone.html
- HyperLoom https://code.it4i.cz/ADAS/loom https://code.it4i.cz/ADAS/loom
- Dask http://dask.pydata.org/en/latest/ https://github.com/dask/dask
- Stimela https://github.com/SpheMakh/Stimela https://github.com/SpheMakh/Stimela/wiki https://www.acru.ukzn.ac.za/~cosmosafari2017/wp-content/uploads/2017/02/makhathini.pdf
- JTracker https://jtracker.io/ https://github.com/jtracker-io
- PipelineDog http://pipeline.dog/ https://github.com/zhouanbo/pipelinedog https://doi.org/10.1093/bioinformatics/btx759
- DALiuGE https://arxiv.org/abs/1702.07617 https://github.com/ICRAR/daliuge https://daliuge.readthedocs.io/
- Overseer https://github.com/framed-data/overseer
- Squonk https://squonk.it/
- GC3Pie https://github.com/uzh/gc3pie
- Fractalide https://github.com/fractalide/fractalide
- TOGGLe https://doi.org/10.1101/245480 http://toggle.southgreen.fr/
- Askalon http://www.askalon.org
- Eclipse ICE (The Integrated Computational Environment) https://www.eclipse.org/ice
- Sandia Analysis Workbench (SAW) http://www.sandia.gov/saw/
- dispel4py https://github.com/dispel4py/dispel4py
- Jobber https://pypi.python.org/pypi/Jobber/0.1.4
- NeatSeq-Flow http://neatseq-flow.readthedocs.io/
- S4M https://bitbucket.org/uqokorn/s4m_base/wiki/Home
- Loom http://med.stanford.edu/gbsc/loom.html https://github.com/StanfordBioinformatics/loom http://loom.readthedocs.io/en/latest/templates.html
- Watchdog https://doi.org/10.1186/s12859-018-2107-4 https://github.com/klugem/watchdog
- phpflo https://github.com/phpflo/phpflo
- BASTet: Berkeley Analysis and Storage Toolkit https://openmsi.nersc.gov/openmsi/client/bastet.html https://biorack.github.io/BASTet/ https://doi.org/10.1109/TVCG.2017.2744479
- Tavaxy: Pattern based workflow system for the bioinformatics domain http://www.tavaxy.org/
- Ginflow: Decentralised adaptive workflow engine https://ginflow.inria.fr/
- SciApps: A cloud-based platform for reproducible bioinformatics workflows https://doi.org/10.1093/bioinformatics/bty439 https://www.sciapps.org/
- Stoa: Script Tracking for Observational Astronomy https://github.com/petehague/Stoa
- Collective Knowledge (CK) framework http://cknowledge.org/
- QosCosGrid (QCG) http://www.qoscosgrid.org/ http://www.qoscosgrid.org/trac/qcg-broker/wiki/qcg-advanced-client%20
- High-Throughput Binding Affinity Calculator (HTBAC) https://htbac.readthedocs.io/en/latest/ https://github.com/radical-cybertools/htbac https://arxiv.org/abs/1801.01174
- BioWorkbench (Swift-based) https://arxiv.org/abs/1801.03915 https://github.com/mmondelli/bioworkbench
- ENVI Task Engine https://gbdxdocs.digitalglobe.com/docs/envi-task-engine https://www.harrisgeospatial.com/Learn/Whitepapers/TabId/2359/ArtMID/10212/ArticleID/17299/Workflow-Tools-in-ENVI.aspx https://envi-py-engine.readthedocs.io/en/latest/index.html
- Pypeline https://github.com/cgarciae/pypeln
- mpipe http://vmlaker.github.io/mpipe/
- idseq-dag https://github.com/chanzuckerberg/idseq-dag
- Piper (based upon GATK Queue) https://github.com/NationalGenomicsInfrastructure/piper
- Apache Object Oriented Data Technology (OODT) http://oodt.apache.org/
- JX Workflow (DSL for Makeflow) https://ccl.cse.nd.edu/software/manuals/jx-quick.html http://ccl.cse.nd.edu/research/papers/jx-escience-2018.pdf
- The Adaptable IO System (ADIOS), ADIOS using applications can be the orchestrated into a workflow http://csmd.ornl.gov/adios
- GenPipes https://doi.org/10.1101/459552 https://bitbucket.org/mugqic/genpipes/src/master/
- Argo https://argoproj.github.io/ https://github.com/argoproj/argo https://github.com/argoproj/argo/blob/master/examples/README.md
- Reana https://reana.readthedocs.io/en/latest/ https://github.com/reanahub/reana
- Cuisine Framework https://www.astron.nl/~renting/cuisine.html
- Niassa https://github.com/oicr-gsi/niassa https://oicr-gsi.github.io/niassa-docs/
- pypeFLOW https://github.com/PacificBiosciences/pypeFLOW
- Tiny Cloud Engine http://ka.cb.k.u-tokyo.ac.jp/tce/
- Xbowflow https://github.com/ChrisSuess/Project-Xbow/tree/master/xbowflow
- AdaptiveMd https://github.com/markovmodel/adaptivemd
- Meshroom https://github.com/alicevision/meshroom
- LSST Data Management https://github.com/lsst/pipe_base
- CGAT-core https://github.com/cgat-developers/cgat-core
- Prefect https://docs.prefect.io/
- Apache SCXML engine https://commons.apache.org/proper/commons-scxml/guide/core-engine.html https://commons.apache.org/proper/commons-scxml/guide/scxml-documents.html
- IceProd https://github.com/WIPACrepo/iceprod
- AnADAMA2 http://huttenhower.sph.harvard.edu/anadama2
- Luna https://luna-lang.org/
- Passerelle https://code.google.com/archive/a/eclipselabs.org/p/passerelle
- Kurator-Akka https://github.com/kurator-org/kurator-akka
- Jug http://doi.org/10.5334/jors.161
- Node-RED https://nodered.org/
- Databolt Flow https://github.com/d6t/d6tflow
- Frictionless Data Package Pipelines https://github.com/frictionlessdata/datapackage-pipelines
- DataFlows https://github.com/datahq/dataflows
- Volcano https://github.com/volcano-sh/volcano
- DataJoint https://datajoint.io/
- DIRAC3 (LHCb grid software) https://iopscience.iop.org/article/10.1088/1742-6596/219/6/062029
原文:https://github.com/common-workflow-language/common-workflow-language/wiki/Existing-Workflow-systems
本文:
讨论:请加入知识星球或者小红圈【首席架构师圈】
- 98 次浏览
SEO Title
Computational Data Analysis Workflow Systems