Storing Nagios Log/Elasticsearch Data to S3?

Software & Operating Systems
1

Bit of a long shot, but what gives.
I’m in the process of setting up a Log Server. It’ll be Nagios Log (i’ve had a longer thread on that subject).

Now i realized that, with just one Windows Server and 6 Linux Servers, i’m generating around 1.5G of Data per Day. Since, in it’s final form, there will be around 700 Systems reporting to it (maybe more depending on Firewalls, switches, UPS’s etc…), This could go up to 200G per day. With a minimum of 30 days to keep, depending on need maybe longer, we’re looking at 10TB+ of data.

I’d really rather not make a big ass virtual Disk and throw all the Data on there. Recovering this from Backup would be a nightmare and our Virtual environment (while plenty) is limited in Diskspace. It’s possible, but i’d like to avoid that.
My company also hosts S3 Storage for internal and customer use. There’s several 100s of Terabytes of storage that we could use for that. So, is there a way to get Nagios Logserver/Elasticsearch to actually put production Data on S3 that still can be actively used? All i found is replicating to S3 for backup.

I’d be fine with a “tiered” setup. Like, keep a week of Data on the Server and move the rest to S3. As long as i can still use that Data in Dashboards and such.

Or would somehow mounting S3 storage to the Server work here? I’m sorry if this sounds dumb, but my Experience with S3 storage is really, limited.
Any input on that subject would be highly appreciated as the Nagios Documentation doesn’t really help with this.

2

I’m more of an Azure person but I think this is what you would need to be able to present your S3 as an SMB or NFS share:

https://aws.amazon.com/storagegateway/file/

1 Like
3

Thats nice. I also just found out about S3FS:


In both cases, i’d be mounting my S3 bucket to a local folder to save the Data.
I was hoping that there would be some kind of feature in Elasticsearch or Nagios Log for that matter, that would natively integrate with S3 rather than going through a local mount.

4

Yes, I’m not familiar enough with Elasticsearch & Nagios to know how much support they support the S3 API. Enough to monitor S3 I guess, but maybe not enough to use it as a repository for logs?