XML Tutorials

PHP SimpleXML Tutorial
View tutorial ...
 
PHP DOM
View tutorial ...
 
PHP XML_Parser Parsing RSS Feed tutorial
View tutorial ...
 

Apache Tutorials

Apache mod_expires
View tutorial ...
 
Simple Apache Httpd Server Setup Tutorial
View tutorial ...
 
Apache Module Context
View tutorial ...
 

JQuery

JQUERY Bar Chart and CSS layout example
View tutorial ...
 
JQUERY Plugin Bar Chart
View tutorial ...
 

Ext Tutorials

EXT Select Group List Tutorial
View tutorial ...
 
EXT Form.FieldSet Example
View tutorial ...
 
EXT Attaching Ajax to Ext:Panel
View tutorial ...
 

Apache Flume spooldir Setup tutorial

In this tutorial, I will show how to setup Apache Flume Sink SpoolDir to hadoop hdfs remotely (on the same subnet).

Step 1. create flume config agent file for source, channel and sink.
### create a file named /etc/flume/conf/spool_dir.conf
### /var/log/flumeSpool is the spool directory
### where I will push file in after starting agent

spool_dir.sources = src-1
spool_dir.channels = channel-1
spool_dir.sinks = sink_to_hdfs

# source
spool_dir.sources.src-1.type = spooldir
spool_dir.sources.src-1.channels = channel-1
spool_dir.sources.src-1.spoolDir = /var/log/flumeSpool
spool_dir.sources.src-1.fileHeader = true
# HDFS sinks
spool_dir.sinks.sink_to_hdfs.type = hdfs
spool_dir.sinks.sink_to_hdfs.hdfs.fileType = DataStream
# change to your host
spool_dir.sinks.sink_to_hdfs.hdfs.path = hdfs://192.168.1.20/flume/remotetest
spool_dir.sinks.sink_to_hdfs.hdfs.filePrefix = customevent
spool_dir.sinks.sink_to_hdfs.hdfs.fileSuffix = .log
spool_dir.sinks.sink_to_hdfs.hdfs.batchSize = 1000


# Use a channel which buffers events in memory
spool_dir.channels.channel-1.type = file
spool_dir.channels.channel-1.checkpointDir = /var/flume/spool_dir_checkpoint
spool_dir.channels.channel-1.dataDirs = /var/flume/spool_dir_data

# Bind the source and sink to the channel
spool_dir.sources.src-1.channels = channel-1
spool_dir.sinks.sink_to_hdfs.channel = channel-1

Step 2. Start the flume-ng agent
if JAVA_HOME not defined
# export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java
assuming flume-ng in the path
# flume-ng agent -c /etc/flume/conf \
-f /etc/flume/conf/spool_dir.conf -n spool_dir

Tail the log file to make sure no error.
# tail -f /var/log/flume/flume.log 


Step 3. copy file and check result on remote server
### I will copy all apache logs to the spool
cp /var/log/httpd/access.* /var/log/flumeSpool/

### you should see lots of activities in the flume log
### now check hdfs server
### login to the hdfs server
# hadoop fs -ls /flume/remotetest
 
 

MYSQL Tutorials

MYSQL Grants Administration
View tutorial ...
 
MYSQL stored procedure cursor
View tutorial ...
 
MYSQL stored procedure IN Parameter Array Data
View tutorial ...
 
MYSQL Trigger Tutorial audit trail solution
View tutorial ...
 
MYSQL Replication
View tutorial ...
 
MYSQL stored procedure cursor tutorial - calculating simple moving average
View tutorial ...
 
mysql_install_db Howto
View tutorial ...
 
MYSQL function tutorial
View tutorial ...
 
MYSQL stored procedure tutorial In Parameter Example
View tutorial ...
 
MYSQL Database Server Setup Tutorial
View tutorial ...
 

PHP Tutorials

PHP XML_Parser Parsing RSS Feed tutorial
View tutorial ...
 
OOP Inheritance overriding DOMDOCUMENT
View tutorial ...
 
Php Recursive Function Retrieve Tree Structure
View tutorial ...
 
PHP SimpleXML Tutorial
View tutorial ...
 
PHP memcache Tutorial
View tutorial ...
 
PHP Caching - Cache_Lite Tutorial
View tutorial ...
 
PHP DOM
View tutorial ...
 
PHP calling MYSQL stored procedure
View tutorial ...
 
Php tutorial SEO database driven website
View tutorial ...
 
Howto Symfony tutorial mapping existing database table
View tutorial ...
 
PHP REST Web Service tutorial -- ebay web service API tutorial
View tutorial ...
 
DB Abstration Layer Class in PHP
View tutorial ...
 
Custom Session Handler in PHP
View tutorial ...
 
JSON AJAX JQUERY
View tutorial ...
 
PHP JSON AJAX
View tutorial ...
 
@ 2012 LAMPDEV.ORG & Terms of Use

Notice: Undefined index: metadesc in /home/toyswhol/zf_program/application/controllers/IndexController.php on line 107