https://github.com/fluent/fluent-plugin-s3
Amazon S3 input and output plugin for Fluentd
https://github.com/fluent/fluent-plugin-s3
Keywords
aws fluentd fluentd-plugin s3
Keywords from Contributors
cncf data-collector activerecord mvc activejob rubygems msgpack rack crash-reporting nosql
Last synced: about 5 hours ago
JSON representation
Repository metadata
Amazon S3 input and output plugin for Fluentd
- Host: GitHub
- URL: https://github.com/fluent/fluent-plugin-s3
- Owner: fluent
- Created: 2011-09-28T05:24:03.000Z (over 14 years ago)
- Default Branch: master
- Last Pushed: 2025-12-02T01:36:00.000Z (about 1 month ago)
- Last Synced: 2025-12-13T14:47:03.114Z (29 days ago)
- Topics: aws, fluentd, fluentd-plugin, s3
- Language: Ruby
- Homepage: https://docs.fluentd.org/output/s3
- Size: 648 KB
- Stars: 320
- Watchers: 23
- Forks: 217
- Open Issues: 24
- Releases: 0
-
Metadata Files:
- Readme: README.md
- Changelog: ChangeLog
- Authors: AUTHORS
README.md
Amazon S3 plugin for Fluentd
Overview
s3 output plugin buffers event logs in local file and upload it to S3
periodically.
This plugin splits files exactly by using the time of event logs (not the time
when the logs are received). For example, a log '2011-01-02 message B' is
reached, and then another log '2011-01-03 message B' is reached in this order,
the former one is stored in "20110102.gz" file, and latter one in
"20110103.gz" file.
s3 input plugin reads data from S3 periodically. This plugin uses
SQS queue on the region same as S3 bucket.
We must setup SQS queue and S3 event notification before use this plugin.
⚠️ Be sure to keep a close eye on S3 costs, as a few user have reported unexpectedly high costs.
Requirements
| fluent-plugin-s3 | fluentd | ruby |
|---|---|---|
| >= 1.0.0 | >= v0.14.0 | >= 2.1 |
| < 1.0.0 | >= v0.12.0 | >= 1.9 |
Installation
Simply use RubyGems:
# install latest version
$ gem install fluent-plugin-s3 --no-document # for fluentd v1.0 or later
# If you need to install specifiv version, use -v option
$ gem install fluent-plugin-s3 -v 1.3.0 --no-document
# For v0.12. This is for old v0.12 users. Don't use v0.12 for new deployment
$ gem install fluent-plugin-s3 -v "~> 0.8" --no-document # for fluentd v0.12
Configuration: credentials
Both S3 input/output plugin provide several credential methods for authentication/authorization.
See Configuration: credentials about details.
Output Plugin
See Configuration: Output about details.
Input Plugin
See Configuration: Input about details.
Tips and How to
Migration guide
See Migration guide from v0.12 about details.
Website, license, et. al.
| Web site | http://fluentd.org/ |
|---|---|
| Documents | http://docs.fluentd.org/ |
| Source repository | http://github.com/fluent/fluent-plugin-s3 |
| Discussion | http://groups.google.com/group/fluentd |
| Author | Sadayuki Furuhashi |
| Copyright | (c) 2011 FURUHASHI Sadayuki |
| License | Apache License, Version 2.0 |
Owner metadata
- Name: Fluentd: Unified Logging Layer
- Login: fluent
- Email:
- Kind: organization
- Description: Fluentd is a cloud native logging solution to unify data collection and consumption.
- Website: http://www.fluentd.org/
- Location: Cloud Native Computing Foundation
- Twitter:
- Company:
- Icon url: https://avatars.githubusercontent.com/u/859518?v=4
- Repositories: 89
- Last ynced at: 2023-04-10T10:21:17.081Z
- Profile URL: https://github.com/fluent
GitHub Events
Total
- Issues event: 11
- Watch event: 5
- Delete event: 7
- Issue comment event: 39
- Push event: 13
- Pull request review comment event: 26
- Pull request review event: 34
- Pull request event: 28
- Fork event: 6
- Create event: 7
Last Year
- Issues event: 3
- Watch event: 2
- Delete event: 3
- Issue comment event: 15
- Push event: 5
- Pull request review event: 2
- Pull request review comment event: 2
- Pull request event: 8
- Fork event: 2
- Create event: 3
Committers metadata
Last synced: 1 day ago
Total Commits: 541
Total Committers: 95
Avg Commits per committer: 5.695
Development Distribution Score (DDS): 0.63
Commits in past year: 6
Committers in past year: 3
Avg Commits per committer in past year: 2.0
Development Distribution Score (DDS) in past year: 0.5
| Name | Commits | |
|---|---|---|
| Masahiro Nakagawa | r****y@g****m | 200 |
| Kenji Okimoto | o****o@c****m | 117 |
| FURUHASHI Sadayuki | f****i@u****p | 14 |
| sonots | s****s@g****m | 14 |
| Kentaro Hayashi | h****i@c****m | 12 |
| Takuro Ashie | a****e@c****m | 12 |
| Michael H. Oshita | i****b@g****m | 10 |
| Daijiro Fukuda | f****a@c****m | 10 |
| Hiroshi Hatake | c****c@g****m | 7 |
| Zohar Arad | z****r@z****m | 7 |
| tpetr | t****r@h****m | 7 |
| dependabot[bot] | 4****] | 6 |
| Jordan Hamel | j****l@g****m | 5 |
| riquemon | 5****n | 4 |
| yannicktekulve | y****e@t****l | 3 |
| kanga333 | e****w@g****m | 3 |
| funtusov | s****s@g****m | 3 |
| Joshua Atkins | j****h@a****m | 3 |
| shunsuke maeda | d****3@g****m | 3 |
| Yuichi Tateno | h****h@g****m | 3 |
| Kazuki Ohta | k****a@g****m | 3 |
| KUOKA Yusuke | y****a@g****m | 3 |
| Jens Raaby | g****t@j****m | 3 |
| Gowtam Lal | g****l@h****m | 2 |
| Naoya Ito | i****a@g****m | 2 |
| Peter Gillard-Moss | p****s@g****m | 2 |
| Takumi Sakamoto | t****a@g****m | 2 |
| cmarodrigues | c****s@g****m | 2 |
| skizot722 | s****i@g****m | 2 |
| Greg Symons | g****s@a****m | 2 |
| and 65 more... | ||
Committer domains:
- clear-code.com: 5
- hubspot.com: 2
- jp.sony.com: 1
- ktamura.com: 1
- qq.com: 1
- bulknews.net: 1
- titech.ac: 1
- woinar.de: 1
- citywire.co.uk: 1
- focuscura.nl: 1
- fastretailing.com: 1
- cookpad.com: 1
- scentregroup.com: 1
- affinipay.com: 1
- jensraaby.com: 1
- ascendantcom.com: 1
- topicus.nl: 1
- zohararad.com: 1
- users.sourceforge.jp: 1
- mail.coder-hugo.de: 1
- junyks.cz: 1
- rezonate.io: 1
- gelens.org: 1
- jotes.work: 1
- fcg.fyi: 1
- arrakis.se: 1
- squareup.com: 1
- corryh.com: 1
- mobfox.com: 1
- unicorn.inc: 1
- ip-192-168-11-229.us-west-2.compute.internal: 1
- cisco.com: 1
- tadaweb.com: 1
- htc.com: 1
- p-lucky.net: 1
- thebackplane.com: 1
- mx.itmedia.co.jp: 1
- gnnk.net: 1
Issue and Pull Request metadata
Last synced: 18 days ago
Total issues: 82
Total pull requests: 69
Average time to close issues: over 1 year
Average time to close pull requests: 6 months
Total issue authors: 75
Total pull request authors: 27
Average comments per issue: 4.44
Average comments per pull request: 2.09
Merged pull request: 51
Bot issues: 0
Bot pull requests: 9
Past year issues: 0
Past year pull requests: 12
Past year average time to close issues: N/A
Past year average time to close pull requests: 2 days
Past year issue authors: 0
Past year pull request authors: 3
Past year average comments per issue: 0
Past year average comments per pull request: 0.0
Past year merged pull request: 9
Past year bot issues: 0
Past year bot pull requests: 5
Top Issue Authors
- DivyaYash (2)
- tgilman (2)
- amitdhawan (2)
- atkinsj (2)
- daipom (2)
- bhaveshmodiatekaplus (2)
- valentinacala (2)
- nilyin (1)
- anarwal (1)
- tkavak (1)
- TomMizrachi (1)
- mehtabhassan (1)
- jottos (1)
- ps-e (1)
- hlx98007 (1)
Top Pull Request Authors
- daipom (10)
- kenhys (10)
- dependabot[bot] (9)
- repeatedly (4)
- ashie (4)
- Watson1978 (4)
- ddukbg (4)
- funtusov (2)
- mufazzal (2)
- joe-pll (2)
- rmontenegroo (2)
- chancez (1)
- khetman (1)
- JustinHead (1)
- islue (1)
Top Issue Labels
- stale (45)
- enhancement (14)
- waiting-for-triage (6)
- waiting-for-user (6)
- incomplete (5)
- invalid (4)
- bug (4)
- help wanted (3)
Top Pull Request Labels
- dependencies (9)
- enhancement (7)
- stale (7)
- github_actions (5)
Package metadata
- Total packages: 4
-
Total downloads:
- rubygems: 292,432,037 total
- Total docker downloads: 1,966,477,952
- Total dependent packages: 5 (may contain duplicates)
- Total dependent repositories: 733 (may contain duplicates)
- Total versions: 194
- Total maintainers: 11
gem.coop: fluent-plugin-s3
Amazon S3 output plugin for Fluentd event collector
- Homepage: https://github.com/fluent/fluent-plugin-s3
- Documentation: http://www.rubydoc.info/gems/fluent-plugin-s3/
- Licenses: Apache-2.0
- Latest release: 1.8.3 (published 11 months ago)
- Last Synced: 2026-01-08T04:00:31.360Z (3 days ago)
- Versions: 96
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 146,173,849 Total
- Docker Downloads: 983,238,976
-
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 0.064%
- Docker downloads count: 0.09%
- Downloads: 0.167%
- Maintainers (10)
rubygems.org: fluent-plugin-s3
Amazon S3 output plugin for Fluentd event collector
- Homepage: https://github.com/fluent/fluent-plugin-s3
- Documentation: http://www.rubydoc.info/gems/fluent-plugin-s3/
- Licenses: Apache-2.0
- Latest release: 1.8.3 (published 11 months ago)
- Last Synced: 2026-01-09T13:55:41.220Z (1 day ago)
- Versions: 96
- Dependent Packages: 5
- Dependent Repositories: 733
- Downloads: 146,252,924 Total
- Docker Downloads: 983,238,976
-
Rankings:
- Docker downloads count: 0.125%
- Downloads: 0.157%
- Dependent repos count: 1.266%
- Average: 1.568%
- Forks count: 1.855%
- Dependent packages count: 2.811%
- Stargazers count: 3.192%
- Maintainers (10)
gem.coop: fluent-plugin-s3-file-inclusion
Amazon S3 output plugin for Fluentd event collector
- Homepage: https://github.com/fluent/fluent-plugin-s3
- Documentation: http://www.rubydoc.info/gems/fluent-plugin-s3-file-inclusion/
- Licenses: Apache-2.0
- Latest release: 1.4.1 (published about 5 years ago)
- Last Synced: 2026-01-08T04:17:17.061Z (3 days ago)
- Versions: 1
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 2,632 Total
-
Rankings:
- Dependent repos count: 0.0%
- Dependent packages count: 0.0%
- Average: 30.228%
- Downloads: 90.685%
- Maintainers (1)
rubygems.org: fluent-plugin-s3-file-inclusion
Amazon S3 output plugin for Fluentd event collector
- Homepage: https://github.com/fluent/fluent-plugin-s3
- Documentation: http://www.rubydoc.info/gems/fluent-plugin-s3-file-inclusion/
- Licenses: Apache-2.0
- Latest release: 1.4.1 (published about 5 years ago)
- Last Synced: 2026-01-08T04:17:17.038Z (3 days ago)
- Versions: 1
- Dependent Packages: 0
- Dependent Repositories: 0
- Downloads: 2,632 Total
-
Rankings:
- Forks count: 1.654%
- Stargazers count: 2.978%
- Dependent packages count: 15.706%
- Average: 32.602%
- Dependent repos count: 46.782%
- Downloads: 95.893%
- Maintainers (1)
Dependencies
- rake >= 0.9.2 development
- rexml >= 0 development
- test-unit >= 3.0.8 development
- test-unit-rr >= 1.0.3 development
- timecop >= 0 development
- aws-sdk-s3 ~> 1.60
- aws-sdk-sqs ~> 1.23
- fluentd >= 0.14.22, < 2
- actions/checkout v3 composite
- ruby/setup-ruby v1 composite
- actions/stale v7 composite
Score: 31.932667236176364