Hey guys/girls:
Starting here: http://docs.graylog.org/en/1.3/pages/architecture.html
You see the image for the minimal setup (which is how their OVA is setup) all in one system.
I am more interested in the bigger production setup ( clustered elasticsearch ... etc)
But I havent found in the docs or anywhere online about setting this up. And of course the docs are missing much of the setup details for breaking the roles out onto separate machines.
I figured I would start with trying to split the DBs out to one machine and the graylog stuff on the other.
From what I can tell, I finally figured out how to connect the graylog-server to the mongoDB (not from info in the doc pages) and it looks like it it connecting fine to the ES DB.
The rest api is (from what I am seeing) the only thing not working
The graylog-server log.
2016-04-01T20:11:01.721-04:00 INFO [CmdLineTool] Loaded plugins: [Anonymous Usage Statistics 1.1.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]]
2016-04-01T20:11:01.801-04:00 INFO [MongoDbConfiguration] You're using deprecated configuration options for MongoDB. Please use mongodb_uri.
2016-04-01T20:11:01.837-04:00 INFO [MongoDbConfiguration] Suggested value for mongodb_uri = mongodb://10.2.3.13:27017/graylog2
2016-04-01T20:11:01.879-04:00 INFO [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dlog4j.configuration=file:///etc/graylog/server/log4j.xml
2016-04-01T20:11:05.688-04:00 INFO [InputBufferImpl] Message journal is enabled.
2016-04-01T20:11:05.990-04:00 INFO [LogManager] Loading logs.
2016-04-01T20:11:06.098-04:00 INFO [LogManager] Logs loading complete.
2016-04-01T20:11:06.098-04:00 INFO [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2016-04-01T20:11:06.126-04:00 INFO [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2016-04-01T20:11:06.531-04:00 INFO [NodeId] Node ID: 351da076-728c-4337-ae3c-8e83292a4979
2016-04-01T20:11:06.759-04:00 INFO [node] [graylog2-server] version[1.7.1], pid[3804], build[b88f43f/2015-07-29T09:54:16Z]
2016-04-01T20:11:06.760-04:00 INFO [node] [graylog2-server] initializing ...
2016-04-01T20:11:06.881-04:00 INFO [plugins] [graylog2-server] loaded [graylog-monitor], sites []
2016-04-01T20:11:10.055-04:00 INFO [node] [graylog2-server] initialized
2016-04-01T20:11:10.338-04:00 INFO [Version] HV000001: Hibernate Validator 5.1.3.Final
2016-04-01T20:11:10.511-04:00 INFO [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-04-01T20:11:12.895-04:00 INFO [RulesEngineProvider] No static rules file loaded.
2016-04-01T20:11:12.962-04:00 INFO [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-04-01T20:11:14.261-04:00 INFO [ServerBootstrap] Graylog server 1.2.2 (91c7822) starting up. (JRE: Oracle Corporation 1.8.0_77 on Linux 2.6.32-573.22.1.el6.x86_64)
2016-04-01T20:11:14.321-04:00 INFO [PeriodicalsService] Starting 23 periodicals ...
2016-04-01T20:11:14.336-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2016-04-01T20:11:14.337-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2016-04-01T20:11:14.340-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2016-04-01T20:11:14.352-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], polling every [20s].
2016-04-01T20:11:14.353-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2016-04-01T20:11:14.343-04:00 INFO [node] [graylog2-server] starting ...
2016-04-01T20:11:14.372-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.DeadLetterThread] periodical, running forever.
2016-04-01T20:11:14.374-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2016-04-01T20:11:14.375-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2016-04-01T20:11:14.376-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2016-04-01T20:11:14.377-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2016-04-01T20:11:14.379-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2016-04-01T20:11:14.382-04:00 INFO [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2016-04-01T20:11:14.382-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.StreamThroughputCounterManagerThread] periodical in [0s], polling every [1s].
2016-04-01T20:11:14.392-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2016-04-01T20:11:14.419-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2016-04-01T20:11:14.422-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2016-04-01T20:11:14.431-04:00 INFO [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [300s].
2016-04-01T20:11:14.431-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2016-04-01T20:11:14.436-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2016-04-01T20:11:14.437-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2016-04-01T20:11:14.502-04:00 WARN [ClusterConfigServiceImpl] Couldn't find cluster config of type org.graylog2.periodical.IndexRangesMigrationPeriodical.MongoIndexRangesMigrationComplete
2016-04-01T20:11:14.503-04:00 INFO [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2016-04-01T20:11:14.602-04:00 INFO [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2016-04-01T20:11:14.602-04:00 INFO [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2016-04-01T20:11:14.611-04:00 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2016-04-01T20:11:14.611-04:00 INFO [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2016-04-01T20:11:14.841-04:00 INFO [Reflections] Reflections took 413 ms to scan 1 urls, producing 2 keys and 2 values
2016-04-01T20:11:14.859-04:00 ERROR [ServiceManager] Service IndexerSetupService [FAILED] has failed in the STARTING state.
org.elasticsearch.transport.BindTransportException: Failed to bind to [9350]
at org.elasticsearch.transport.netty.NettyTransport.bindServerBootstrap(NettyTransport.java:422)
at org.elasticsearch.transport.netty.NettyTransport.doStart(NettyTransport.java:283)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.transport.TransportService.doStart(TransportService.java:153)
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:85)
at org.elasticsearch.node.internal.InternalNode.start(InternalNode.java:257)
at org.graylog2.initializers.IndexerSetupService.startUp(IndexerSetupService.java:114)
at com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54)
at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.common.netty.channel.ChannelException: Failed to bind to: /10.2.3.13:9350
at org.elasticsearch.common.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
at org.elasticsearch.transport.netty.NettyTransport$1.onPortNumber(NettyTransport.java:413)
at org.elasticsearch.common.transport.PortsRange.iterate(PortsRange.java:58)
at org.elasticsearch.transport.netty.NettyTransport.bindServerBootstrap(NettyTransport.java:409)
... 9 more
Caused by: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.elasticsearch.common.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
... 1 more
2016-04-01T20:11:14.894-04:00 ERROR [InputSetupService] Not starting any inputs because lifecycle is: Uninitialized [LB:DEAD]
2016-04-01T20:11:14.895-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.AlertScannerThread].
2016-04-01T20:11:14.895-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.AlertScannerThread] complete, took <0ms>.
2016-04-01T20:11:14.896-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread].
2016-04-01T20:11:14.896-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] complete, took <0ms>.
2016-04-01T20:11:14.897-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ClusterHealthCheckThread].
2016-04-01T20:11:14.900-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ClusterHealthCheckThread] complete, took <0ms>.
2016-04-01T20:11:14.900-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexerClusterCheckerThread].
2016-04-01T20:11:14.900-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexerClusterCheckerThread] complete, took <0ms>.
2016-04-01T20:11:14.900-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRetentionThread].
2016-04-01T20:11:14.900-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRetentionThread] complete, took <0ms>.
2016-04-01T20:11:14.900-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.IndexRotationThread].
2016-04-01T20:11:14.900-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.IndexRotationThread] complete, took <0ms>.
2016-04-01T20:11:14.900-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.VersionCheckThread].
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.VersionCheckThread] complete, took <0ms>.
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.ThrottleStateUpdaterThread].
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.ThrottleStateUpdaterThread] complete, took <0ms>.
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventPeriodical].
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventPeriodical] complete, took <0ms>.
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.events.ClusterEventCleanupPeriodical].
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.events.ClusterEventCleanupPeriodical] complete, took <0ms>.
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog2.periodical.PurgeExpiredCollectorsThread].
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog2.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2016-04-01T20:11:14.901-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical].
2016-04-01T20:11:14.902-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] complete, took <0ms>.
2016-04-01T20:11:14.902-04:00 INFO [PeriodicalsService] Shutting down periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical].
2016-04-01T20:11:14.902-04:00 INFO [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] complete, took <0ms>.
2016-04-01T20:11:14.919-04:00 WARN [BufferSynchronizerService] Elasticsearch is unavailable. Not waiting to clear buffers and caches, as we have no healthy cluster.
2016-04-01T20:11:14.908-04:00 INFO [LogManager] Shutting down.
2016-04-01T20:11:14.925-04:00 INFO [OutputSetupService] Stopping output org.graylog2.outputs.BlockingBatchedESOutput
2016-04-01T20:11:14.923-04:00 INFO [JournalReader] Stopping.
2016-04-01T20:11:14.925-04:00 INFO [node] [graylog2-server] stopping ...
2016-04-01T20:11:14.968-04:00 INFO [node] [graylog2-server] stopped
2016-04-01T20:11:14.968-04:00 INFO [node] [graylog2-server] closing ...
2016-04-01T20:11:14.997-04:00 INFO [node] [graylog2-server] closed
2016-04-01T20:11:15.073-04:00 INFO [LogManager] Shutdown complete.
2016-04-01T20:11:22.686-04:00 INFO [RestApiService] Adding security context factory: <org.graylog2.security.ShiroSecurityContextFactory@497babef>
2016-04-01T20:11:22.717-04:00 INFO [RestApiService] Started REST API at <http://0.0.0.0:12900/>
2016-04-01T20:11:22.722-04:00 INFO [RestApiService] Shutting down REST API at <http://0.0.0.0:12900/>
2016-04-01T20:11:22.732-04:00 INFO [ServiceManagerListener] Services are now stopped.
2016-04-01T20:11:22.732-04:00 ERROR [ServerBootstrap] Graylog startup failed. Exiting. Exception was:
java.lang.IllegalStateException: Expected to be healthy after starting. The following services are not running: {STARTING=[RestApiService [STARTING]], FAILED=[IndexerSetupService [FAILED]]}
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.checkHealthy(ServiceManager.java:710)
at com.google.common.util.concurrent.ServiceManager$ServiceManagerState.awaitHealthy(ServiceManager.java:535)
at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:301)
at org.graylog2.bootstrap.ServerBootstrap.startCommand(ServerBootstrap.java:114)
at org.graylog2.bootstrap.CmdLineTool.run(CmdLineTool.java:194)
at org.graylog2.bootstrap.Main.main(Main.java:44)
2016-04-01T20:11:22.740-04:00 INFO [Server] SIGNAL received. Shutting down.
2016-04-01T20:11:22.751-04:00 INFO [GracefulShutdown] Graceful shutdown initiated.
2016-04-01T20:11:22.752-04:00 INFO [GracefulShutdown] Node status: [Halting [LB:DEAD]]. Waiting <3sec> for possible load balancers to recognize state change.
2016-04-01T20:11:26.755-04:00 INFO [GracefulShutdown] Goodbye.
Let me know if you have any clues...