https://github.com/grosser/parallel

Ruby: parallel processing made simple and fast
https://github.com/grosser/parallel

Keywords from Contributors

activerecord activejob mvc rubygems code-formatter rubocop static-code-analysis rspec background-jobs rack

Last synced: about 20 hours ago
JSON representation

Repository metadata

Ruby: parallel processing made simple and fast

Host: GitHub
URL: https://github.com/grosser/parallel
Owner: grosser
License: mit
Created: 2009-08-11T18:54:16.000Z (over 16 years ago)
Default Branch: master
Last Pushed: 2025-12-03T19:11:08.000Z (3 months ago)
Last Synced: 2026-02-27T01:47:18.444Z (5 days ago)
Language: Ruby
Size: 807 KB
Stars: 4,232
Watchers: 72
Forks: 257
Open Issues: 35
Releases: 0
Metadata Files:
- Readme: Readme.md
- License: MIT-LICENSE.txt

Parallel

Run any code in parallel Processes(> use all CPUs), Threads(> speedup blocking operations), or Ractors(> use all CPUs).
Best suited for map-reduce or e.g. parallel downloads/uploads.

Install

gem install parallel

Usage

# 2 CPUs -> work in 2 processes (a,b + c)
results = Parallel.map(['a','b','c']) do |one_letter|
  SomeClass.expensive_calculation(one_letter)
end

# 3 Processes -> finished after 1 run
results = Parallel.map(['a','b','c'], in_processes: 3) { |one_letter| SomeClass.expensive_calculation(one_letter) }

# 3 Threads -> finished after 1 run
results = Parallel.map(['a','b','c'], in_threads: 3) { |one_letter| SomeClass.expensive_calculation(one_letter) }

# 3 Ractors -> finished after 1 run
results = Parallel.map(['a','b','c'], in_ractors: 3, ractor: [SomeClass, :expensive_calculation])

Same can be done with each

Parallel.each(['a','b','c']) { |one_letter| ... }

or each_with_index, map_with_index, flat_map

Produce one item at a time with lambda (anything that responds to .call) or Queue.

items = [1,2,3]
Parallel.each( -> { items.pop || Parallel::Stop }) { |number| ... }

Also supports any? or all?

Parallel.any?([1,2,3,4,5,6,7]) { |number| number == 4 }
# => true

Parallel.all?([1,2,nil,4,5]) { |number| number != nil }
# => false

Processes/Threads are workers, they grab the next piece of work when they finish.

Processes

Speedup through multiple CPUs
Speedup for blocking operations
Variables are protected from change
Extra memory used
Child processes are killed when your main process is killed through Ctrl+c or kill -2

Threads

Speedup for blocking operations
Variables can be shared/modified
No extra memory used

Ractors

Ruby 3.0+ only
Speedup for blocking operations
No extra memory used
Very fast to spawn
Experimental and unstable
start and finish hooks are called on main thread
Variables must be passed in Parallel.map([1,2,3].map { |i| [i, ARGV, local_var] }, ...
use Ractor.make_shareable to pass in global objects

ActiveRecord

Connection Lost

Multithreading needs connection pooling, forks need reconnects
Adjust connection pool size in config/database.yml when multithreading

# reproducibly fixes things (spec/cases/map_with_ar.rb)
Parallel.each(User.all, in_processes: 8) do |user|
  user.update_attribute(:some_attribute, some_value)
end
User.connection.reconnect!

# maybe helps: explicitly use connection pool
Parallel.each(User.all, in_threads: 8) do |user|
  ActiveRecord::Base.connection_pool.with_connection do
    user.update_attribute(:some_attribute, some_value)
  end
end

# maybe helps: reconnect once inside every fork
Parallel.each(User.all, in_processes: 8) do |user|
  @reconnected ||= User.connection.reconnect! || true
  user.update_attribute(:some_attribute, some_value)
end

NameError: uninitialized constant

A race happens when ActiveRecord models are autoloaded inside parallel threads
in environments that lazy-load, like development, test, or migrations.

To fix, autoloaded classes before the parallel block with either require '<modelname>' or ModelName.class.

Break

Parallel.map([1, 2, 3]) do |i|
  raise Parallel::Break # -> stops after all current items are finished
end

Parallel.map([1, 2, 3]) { |i| raise Parallel::Break, i if i == 2 } == 2

Kill

Only use if whatever is executing in the sub-command is safe to kill at any point

Parallel.map([1,2,3]) do |x|
  raise Parallel::Kill if x == 1# -> stop all sub-processes, killing them instantly
  sleep 100 # Do stuff
end

Progress / ETA

# gem install ruby-progressbar

Parallel.map(1..50, progress: "Doing stuff") { sleep 1 }

# Doing stuff | ETA: 00:00:02 | ====================               | Time: 00:00:10

Use :finish or :start hook to get progress information.

:start has item and index
:finish has item, index, and result

They are called on the main process and protected with a mutex.
(To just get the index, use the more performant Parallel.each_with_index)

Parallel.map(1..100, finish: -> (item, i, result) { ... do something ... }) { sleep 1 }

Set finish_in_order: true to call the :finish hook in the order of the input (will take longer to see initial output).

Parallel.map(1..9, finish: -> (item, i, result) { puts "#{item} ok" }, finish_in_order: true) { sleep rand }

Worker number

Use Parallel.worker_number to determine the worker slot in which your
task is running.

Parallel.each(1..5, in_processes: 2) { |i| puts "Item: #{i}, Worker: #{Parallel.worker_number}" }
Item: 1, Worker: 1
Item: 2, Worker: 0
Item: 3, Worker: 1
Item: 4, Worker: 0
Item: 5, Worker: 1

Dynamically generating jobs

Example: wait for work to arrive or sleep

queue = []
Thread.new { loop { queue << rand(100); sleep 2 } } # job producer
Parallel.map(Proc.new { queue.pop }, in_processes: 3) { |f| f ? puts("#{f} received") : sleep(1) }

Tips

[Benchmark/Test] Disable threading/forking with in_threads: 0 or in_processes: 0, to run the same code with different setups
[Isolation] Do not reuse previous worker processes: isolation: true
[Stop all processes with an alternate interrupt signal] 'INT' (from ctrl+c) is caught by default. Catch 'TERM' (from kill) with interrupt_signal: 'TERM'
[Process count via ENV] PARALLEL_PROCESSOR_COUNT=16 will use 16 instead of the number of processors detected. This is used to reconfigure a tool using parallel without inserting custom logic.
[Process count] parallel uses a number of processors seen by the OS for process count by default. If you want to use a value considering CPU quota, please add concurrent-ruby to your Gemfile.

TODO

Replace Signal trapping with simple rescue Interrupt handler

Authors

Contributors

Michael Grosser
michael@grosser.it
License: MIT

Owner metadata

Name: Michael Grosser
Login: grosser
Email:
Kind: user
Description: Group Tech Lead at Zendesk and Ruby/Rails/Kubernetes/Golang engineer with too many open-source projects.
Website: grosser.it
Location: San Francisco, CA
Twitter: grosser
Company: Zendesk.com
Icon url: https://avatars.githubusercontent.com/u/11367?v=4
Repositories: 524
Last ynced at: 2024-04-15T13:44:01.129Z
Profile URL: https://github.com/grosser

GitHub Events

Total

Delete event: 1
Pull request event: 5
Fork event: 4
Issues event: 1
Watch event: 95
Issue comment event: 18
Push event: 5
Pull request review event: 37
Pull request review comment event: 35
Create event: 2

Last Year

Delete event: 1
Pull request event: 3
Fork event: 3
Issues event: 1
Watch event: 49
Issue comment event: 17
Push event: 5
Pull request review event: 37
Pull request review comment event: 35
Create event: 2

Committers metadata

Last synced: 1 day ago

Total Commits: 677
Total Committers: 75
Avg Commits per committer: 9.027
Development Distribution Score (DDS): 0.291

Commits in past year: 4
Committers in past year: 1
Avg Commits per committer in past year: 4.0
Development Distribution Score (DDS) in past year: 0.0

Name	Email	Commits
grosser	g**l@g**m	480
Joakim Kolsjö	j**o@g**m	19
Jacob Duffy	d**p@g**m	13
jmozmoz	J**b@g**e	9
yaoguai	l**s@g**m	9
Keiji Yoshimi	w**3@g**m	9
Nathan Broadbent	n****t	9
Yuki INOUE	i**s@g**m	8
Brendan Dougherty	b**y@g**m	7
Guillaume Hain	z**x@z**g	6
Devin Ben-Hur	d**r@w**m	6
Sean Walbran	s**n@s**m	6
Earlopain	1****n	5
Joe Rafaniello	j**e@r**m	4
Takumasa Ochi	a**7@g**m	4
RusHibaM	z**7@s**m	3
Weston Ganger	w**r@g**m	3
Thiago Pradi	t**y@g**m	3
Matt Brennan	m**n@c**m	3
Kenichi Kamiya	k**1@g**m	3
Arlan Jaska	a**a@a**m	3
Bertrand Paquet	b**t@g**m	2
Brian P O'Rourke	b**o@s**t	2
Christian Mayer	g**t@m**m	2
Jurriaan Pruis	e**l@j**l	2
Michael Strüder	m**r@g**m	2
Orien Madgwick	_@o****o	2
Pedro Martinez	2****z	2
Rob Worley	r**y@g**m	2
Vaibhav	v**e@g**m	2
and 45 more...

Committer domains:

Issue and Pull Request metadata

Last synced: 3 days ago

Total issues: 59
Total pull requests: 71
Average time to close issues: 3 months
Average time to close pull requests: 3 months
Total issue authors: 56
Total pull request authors: 31
Average comments per issue: 3.97
Average comments per pull request: 2.13
Merged pull request: 55
Bot issues: 0
Bot pull requests: 0

Past year issues: 2
Past year pull requests: 4
Past year average time to close issues: about 8 hours
Past year average time to close pull requests: about 1 hour
Past year issue authors: 2
Past year pull request authors: 2
Past year average comments per issue: 8.5
Past year average comments per pull request: 3.25
Past year merged pull request: 2
Past year bot issues: 0
Past year bot pull requests: 0

More stats: https://issues.ecosyste.ms/repositories/lookup?url=https://github.com/grosser/parallel

Top Issue Authors

leesanghun6310 (3)
NemyaNation (2)
yeikel (1)
dwery (1)
jfahrenkrug (1)
tveinot (1)
bquorning (1)
michaeleisel (1)
abhinav-94 (1)
bmulholland (1)
owst (1)
florianeck (1)
Minauras (1)
robotdana (1)
eatwithforks (1)

Top Pull Request Authors

grosser (27)
Earlopain (4)
y-yagi (4)
kachick (2)
olleolleolle (2)
takahiro-blab (2)
MITSUBOSHI (2)
duffyjp (2)
RDIL (2)
JasonnnW3000 (2)
pedromartinez (2)
rueyaa332266 (1)
nolanpro (1)
kakra (1)
dleidert (1)

Top Issue Labels

Top Pull Request Labels

Package metadata

Total packages: 17
Total downloads:
- rubygems: 1,448,903,704 total
Total docker downloads: 6,973,827,002
Total dependent packages: 495 (may contain duplicates)
Total dependent repositories: 347,415 (may contain duplicates)
Total versions: 355
Total maintainers: 1

gem.coop: parallel

Run any kind of code in parallel processes

Homepage: https://github.com/grosser/parallel
Documentation: http://www.rubydoc.info/gems/parallel/
Licenses: MIT
Latest release: 1.27.0 (published 11 months ago)
Last Synced: 2026-03-01T23:31:15.911Z (2 days ago)
Versions: 107
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 724,413,353 Total
Docker Downloads: 3,486,913,501
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 0.008%
- Downloads: 0.024%
Maintainers (1)
- grosser

rubygems.org: parallel

Run any kind of code in parallel processes

Homepage: https://github.com/grosser/parallel
Documentation: http://www.rubydoc.info/gems/parallel/
Licenses: MIT
Latest release: 1.27.0 (published 11 months ago)
Last Synced: 2026-03-02T03:32:43.766Z (1 day ago)
Versions: 108
Dependent Packages: 494
Dependent Repositories: 347,415
Downloads: 724,472,383 Total
Docker Downloads: 3,486,913,501
Rankings:
- Downloads: 0.031%
- Dependent repos count: 0.059%
- Docker downloads count: 0.059%
- Dependent packages count: 0.093%
- Stargazers count: 0.317%
- Average: 0.364%
- Forks count: 1.622%
Maintainers (1)
- grosser

proxy.golang.org: github.com/grosser/parallel

Homepage:
Documentation: https://pkg.go.dev/github.com/grosser/parallel#section-documentation
Licenses: mit
Latest release: v1.27.0 (published 11 months ago)
Last Synced: 2026-02-28T23:00:52.544Z (3 days ago)
Versions: 121
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Stargazers count: 1.044%
- Forks count: 1.764%
- Average: 5.796%
- Dependent packages count: 9.576%
- Dependent repos count: 10.802%

gem.coop: grosser-parallel

Run any kind of code in parallel processes

Homepage: http://github.com/grosser/parallel
Documentation: http://www.rubydoc.info/gems/grosser-parallel/
Licenses: mit
Latest release: 0.3.1 (published over 11 years ago)
Last Synced: 2026-02-28T23:00:50.841Z (3 days ago)
Versions: 3
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 8,984 Total
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 16.565%
- Downloads: 49.695%

conda-forge.org: rb-parallel

Homepage: https://rubygems.org/gems/parallel
Licenses: MIT
Latest release: 1.18.0 (published over 6 years ago)
Last Synced: 2026-02-03T23:38:08.780Z (28 days ago)
Versions: 2
Dependent Packages: 1
Dependent Repositories: 0
Rankings:
- Stargazers count: 5.709%
- Forks count: 11.024%
- Average: 21.332%
- Dependent packages count: 28.983%
- Dependent repos count: 39.613%

rubygems.org: grosser-parallel

Run any kind of code in parallel processes

Homepage: http://github.com/grosser/parallel
Documentation: http://www.rubydoc.info/gems/grosser-parallel/
Licenses: mit
Latest release: 0.3.1 (published over 11 years ago)
Last Synced: 2026-02-28T23:00:50.792Z (3 days ago)
Versions: 3
Dependent Packages: 0
Dependent Repositories: 0
Downloads: 8,984 Total
Rankings:
- Stargazers count: 0.271%
- Forks count: 1.475%
- Dependent packages count: 15.706%
- Average: 22.762%
- Dependent repos count: 46.782%
- Downloads: 49.577%

debian-13: ruby-parallel

Homepage: https://github.com/grosser/parallel
Documentation: https://packages.debian.org/trixie/ruby-parallel
Licenses:
Latest release: 1.22.1-3 (published 19 days ago)
Last Synced: 2026-02-13T13:18:15.057Z (18 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%

debian-12: ruby-parallel

Homepage: https://github.com/grosser/parallel
Documentation: https://packages.debian.org/bookworm/ruby-parallel
Licenses:
Latest release: 1.22.1-2 (published 19 days ago)
Last Synced: 2026-02-12T23:37:15.029Z (19 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:

ubuntu-24.10: ruby-parallel

Homepage: https://github.com/grosser/parallel
Licenses:
Latest release: 1.22.1-2 (published 18 days ago)
Last Synced: 2026-02-13T09:54:05.542Z (18 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:

ubuntu-22.04: ruby-parallel

Homepage: https://github.com/grosser/parallel
Licenses:
Latest release: 1.20.1-1 (published 18 days ago)
Last Synced: 2026-02-13T13:22:22.248Z (18 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%

ubuntu-20.04: ruby-parallel

Homepage: https://github.com/grosser/parallel
Licenses:
Latest release: 1.12.1-2 (published 19 days ago)
Last Synced: 2026-02-13T07:19:29.586Z (19 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%

guix: ruby-parallel

Parallel processing in Ruby

Homepage: https://github.com/grosser/parallel
Documentation: https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/packages/ruby-xyz.scm#n13002
Licenses: expat
Latest release: 1.21.0 (published 1 day ago)
Last Synced: 2026-03-02T18:59:12.424Z (1 day ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%

ubuntu-23.10: ruby-parallel

Homepage: https://github.com/grosser/parallel
Licenses: mit
Latest release: 1.22.1-2 (published 18 days ago)
Last Synced: 2026-02-13T18:28:25.212Z (18 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%

ubuntu-23.04: ruby-parallel

Homepage: https://github.com/grosser/parallel
Licenses:
Latest release: 1.22.1-1 (published 21 days ago)
Last Synced: 2026-02-11T06:46:08.121Z (21 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%

debian-10: ruby-parallel

Homepage: https://github.com/grosser/parallel
Documentation: https://packages.debian.org/buster/ruby-parallel
Licenses:
Latest release: 1.12.1-2 (published 20 days ago)
Last Synced: 2026-02-13T04:24:14.987Z (19 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%

ubuntu-24.04: ruby-parallel

Homepage: https://github.com/grosser/parallel
Licenses:
Latest release: 1.22.1-2 (published 25 days ago)
Last Synced: 2026-02-06T15:42:47.439Z (25 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:

debian-11: ruby-parallel

Homepage: https://github.com/grosser/parallel
Documentation: https://packages.debian.org/bullseye/ruby-parallel
Licenses:
Latest release: 1.20.1-1 (published 21 days ago)
Last Synced: 2026-02-13T08:23:18.625Z (18 days ago)
Versions: 1
Dependent Packages: 0
Dependent Repositories: 0
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 100%

Dependencies

.github/workflows/actions.yml actions

actions/checkout v2 composite
ruby/setup-ruby v1 composite
mysql * docker

Gemfile rubygems

mysql2 >= 0 development
activerecord ~> 6.0
bump >= 0
rake >= 0
rspec >= 0
rspec-legacy_formatters >= 0
rspec-rerun >= 0
rubocop >= 0
rubocop-rake >= 0
rubocop-rspec >= 0
ruby-progressbar >= 0
sqlite3 >= 0

Gemfile.lock rubygems

activemodel 6.1.5
activerecord 6.1.5
activesupport 6.1.5
ast 2.4.2
bump 0.10.0
concurrent-ruby 1.1.9
diff-lcs 1.5.0
i18n 1.10.0
minitest 5.15.0
mysql2 0.5.3
parallel 1.22.1
parser 3.1.1.0
rainbow 3.1.1
rake 13.0.6
regexp_parser 2.2.1
rexml 3.2.5
rspec 3.11.0
rspec-core 3.11.0
rspec-expectations 3.11.0
rspec-legacy_formatters 1.0.2
rspec-mocks 3.11.0
rspec-rerun 1.1.0
rspec-support 3.11.0
rubocop 1.26.0
rubocop-ast 1.16.0
rubocop-rake 0.6.0
rubocop-rspec 2.9.0
ruby-progressbar 1.11.0
sqlite3 1.4.2
tzinfo 2.0.4
unicode-display_width 2.1.0
zeitwerk 2.5.4

Score: 35.530396206880965