Jak mogę uzyskać logi zebrane na konsoli za pomocą Flume NG?

Question

Testuję Flume NG (1.2.0) do zbierania dzienników. To prosty test, w którym Flume zbiera plik dziennika flume_test.log i drukuje zebrane logi do konsoli jako sysout. conf/flume.conf jest:

agent.sources = tail 
agent.channels = memoryChannel 
agent.sinks = loggerSink 

agent.sources.tail.type = exec 
agent.sources.tail.command = tail -f /Users/pj/work/flume_test.log 
agent.sources.tail.channels = memoryChannel 

agent.sinks.loggerSink.channel = memoryChannel 
agent.sinks.loggerSink.type = logger 

agent.channels.memoryChannel.type = memory 
agent.channels.memoryChannel.capacity = 100 

I wpadłem koryta jak następuje:

$ $FLUME_HOME/bin/flume-ng agent --conf $FLUME_HOME/conf --conf-file $FLUME_HOME/conf/flume.conf --name agent1 -Dflume.root.logger=DEBUG,console 

Po uruchomieniu dzienniki koryta na konsoli to:

Info: Sourcing environment configuration script /usr/local/lib/flume-ng/conf/flume-env.sh 
+ exec /Library/Java/JavaVirtualMachines/jdk1.7.0_07.jdk/Contents/Home/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -cp '/usr/local/lib/flume-ng/conf:/usr/local/lib/flume-ng/lib/*' -Djava.library.path= org.apache.flume.node.Application --conf-file /usr/local/lib/flume-ng/conf/flume.conf --name agent1 
2012-09-12 18:23:52,049 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1 
2012-09-12 18:23:52,052 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1 
2012-09-12 18:23:52,054 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting 
2012-09-12 18:23:52,056 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 9 
2012-09-12 18:23:52,054 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting 
2012-09-12 18:23:52,056 (lifecycleSupervisor-1-0) [DEBUG - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:191)] Node manager started 
2012-09-12 18:23:52,057 (lifecycleSupervisor-1-1) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:86)] Configuration provider started 
2012-09-12 18:23:52,058 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:188)] Checking file:/usr/local/lib/flume-ng/conf/flume.conf for changes 
2012-09-12 18:23:52,058 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:/usr/local/lib/flume-ng/conf/flume.conf 
2012-09-12 18:23:52,063 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: loggerSink Agent: agent 
2012-09-12 18:23:52,063 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:loggerSink 
2012-09-12 18:23:52,063 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:992)] Created context for loggerSink: type 
2012-09-12 18:23:52,063 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:loggerSink 
2012-09-12 18:23:52,063 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:295)] Starting validation of configuration for agent: agent, initial-configuration: AgentConfiguration[agent] 
SOURCES: {tail={ parameters:{command=tail -f /Users/pj/work/flume_test.log, channels=memoryChannel, type=exec} }} 
CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }} 
SINKS: {loggerSink={ parameters:{type=logger, channel=memoryChannel} }} 

2012-09-12 18:23:52,068 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:450)] Created channel memoryChannel 
2012-09-12 18:23:52,082 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:649)] Creating sink: loggerSink using LOGGER 
2012-09-12 18:23:52,085 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:353)] Post validation configuration for agent 
AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[agent] 
SOURCES: {tail={ parameters:{command=tail -f /Users/pj/work/flume_test.log, channels=memoryChannel, type=exec} }} 
CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }} 
AgentConfiguration created with Configuration stubs for which full validation was performed[agent] 
SINKS: {loggerSink=ComponentConfiguration[loggerSink] 
    CONFIG: 
    CHANNEL:memoryChannel 
} 

2012-09-12 18:23:52,085 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:117)] Channels:memoryChannel 

2012-09-12 18:23:52,085 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:118)] Sinks loggerSink 

2012-09-12 18:23:52,085 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:119)] Sources tail 

2012-09-12 18:23:52,085 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent] 
2012-09-12 18:23:52,085 (conf-file-poller-0) [WARN - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:227)] No configuration found for this host:agent1 

myślę Flume zaczął normalnie, więc umieścić kilka linii do flume_test.log w sposób ciągły. Ale nie drukuje dodanych linii do flume_test.log na konsoli.

Na czym polega problem z tym testem? Dzięki za wszelkie komentarze i poprawki.

Answer 1

Problemem była nazwa niezgodność między nazwą agenta w flume.conf (agent) oraz nazwa agenta po --name (agent1) w skrypcie startowym.

Po zmianie nazwy opcji z --name agent1 na --name agent, problem rozwiązany.

Dzięki za mojego kolegę, Lenny.