Wednesday, December 8, 2010

mkdir

Use this command to create a directory.

   % mkdir essays

To get "into" this directory, do

   % cd essays

To see what files are in essays, do this:

   % ls

There shouldn't be any files there yet, since you just made it. To create files, see cat or emacs.

More is a command used to read text files. For example, we could do this:

   % more poems

The effect of this to let you read the file "poems ". It probably will not fit in one screen, so you need to know how to "turn pages". Here are the basic commands:

q --- quit more
spacebar --- read next page
return key --- read next line
b --- go back one page

For still more information, use the command man more.

mv

Use this command to change the name of file and directories.

   % mv foo foobar

The file that was named foo is now named foobar

ncftp

Use ncftp for anonymous ftp --- that means you don't have to have a password.

   % ncftp ftp.fubar.net

     Connected to ftp.fubar.net

   > get jokes.txt

The file jokes.txt is downloaded from the machine ftp.fubar.net.

print

This is a moderately intelligent print command.

   % print foo

   % print notes.ps

   % print manuscript.dvi

In each case print does the right thing, regardless of whether the file is a text file (like foo ), a postcript file (like notes.ps, or a dvi file (like manuscript.dvi. In these examples the file is printed on the default printer. To see what this is, do

   % print

and read the message displayed. To print on a specific printer, do this:

   % print foo jwb321

   % print notes.ps jwb321

   % print manuscript.dvi jwb321

To change the default printer, do this:

   % setenv PRINTER jwb321

pwd

Use this command to find out what directory you are working in.

   % pwd

/u/ma/jeremy

   % cd homework

   % pwd

/u/ma/jeremy/homework

   % ls

assign-1 assign-2 assign-3

   % cd

   % pwd

/u/ma/jeremy

Jeremy began by working in his "home" directory. Then he cd 'd into his homework subdirectory. Cd means " change directory". He used pwd to check to make sure he was in the right place, then used ls to see if all his homework files were there. (They were). Then he cd'd back to his home directory.

rm

Use rm to remove files from your directory.

   % rm foo

     remove foo? y

   % rm letter*

     remove letter1? y

     remove letter2? y

     remove letter3? n

The first command removed a single file. The second command was intended to remove all files beginning with the string "letter." However, our user (Jeremy?) decided not to remove letter3.

rmdir

Use this command to remove a directory. For example, to remove a directory called "essays", do this:

   % rmdir essays

A directory must be empty before it can be removed. To empty a directory, use rm.

rsh

Use this command if you want to work on a computer different from the one you are currently working on. One reason to do this is that the remote machine might be faster. For example, the command

   % rsh solitude

connects you to the machine solitude. This is one of our public workstations and is fairly fast.
See also: telnet

setenv

   % echo $PRINTER

     labprinter

   % setenv PRINTER myprinter

   % echo $PRINTER

     myprinter

sort

Use this commmand to sort a file. For example, suppose we have a file dict with contents

red rojo

green verde

blue azul

white blanco

black negro

Then we can do this:

   % sort dict

     black negro

     blue azul

     green verde

     red rojo

     white blanco

Here the output of sort went to the screen. To store the output in file we do this:

   % sort dict >dict.sorted

You can check the contents of the file dict.sorted using cat , more , or emacs .

tail

Use this command to look at the tail of a file. For example,

   % head essay.001

displays the last 10 lines of the file essay.001 To see a specific number of lines, do this:

   % head -20 essay.001

This displays the last 20 lines of the file.

tar

Use create compressed archives of directories and files, and also to extract directories and files from an archive. Example:

   % tar -tvzf foo.tar.gz

displays the file names in the compressed archive foo.tar.gz while

   % tar -xvzf foo.tar.gz

extracts the files.

telnet

Use this command to log in to another machine from the machine you are currently working on. For example, to log in to the machine "solitude", do this:

   % telnet solitude

wc

Use this command to count the number of characters, words, and lines in a file. Suppose, for example, that we have a file dict with contents

red rojo

green verde

blue azul

white blanco

black negro

Then we can do this

   % wc dict

     5      10      56 tmp

This shows that dict has 5 lines, 10 words, and 56 characters.
The word count command has several options, as illustrated below:

   % wc -l dict

     5 tmp

   % wc -w dict

     10 tmp

   % wc -c dict

     56 tmp

Unix commands-2

echo

The echo command echoes its arguments. Here are some examples:

   % echo this

     this

   % echo $EDITOR

     /usr/local/bin/emacs

   % echo $PRINTER

     b129lab1

Things like PRINTER are so-called environment variables. This one stores the name of the default printer --- the one that print jobs will go to unless you take some action to change things. The dollar sign before an environment variable is needed to get the value in the variable. Try the following to verify this:

   % echo PRINTER

     PRINTER

ftp

Use ftp to connect to a remote machine, then upload or download files. See also: ncftp
Example 1: We'll connect to the machine fubar.net, then change director to mystuff, then download the file homework11:

   % ftp solitude

     Connected to fubar.net.

     220 fubar.net FTP server (Version wu-2.4(11) Mon Apr 18 17:26:33 MDT 1994) ready.

   Name (solitude:carlson): jeremy

     331 Password required for jeremy.

   Password:

     230 User jeremy logged in.

   ftp> cd mystuff

     250 CWD command successful.

   ftp> get homework11

   ftp> quit

Example 2: We'll connect to the machine fubar.net, then change director to mystuff, then upload the file collected-letters:

   % ftp solitude

     Connected to fubar.net.

     220 fubar.net FTP server (Version wu-2.4(11) Mon Apr 18 17:26:33 MDT 1994) ready.

   Name (solitude:carlson): jeremy

     331 Password required for jeremy.

   Password:

     230 User jeremy logged in.

   ftp> cd mystuff

     250 CWD command successful.

   ftp> put collected-letters

   ftp> quit

The ftp program sends files in ascii (text) format unless you specify binary mode:

   ftp> binary

   ftp> put foo

   ftp> ascii

   ftp> get bar

The file foo was transferred in binary mode, the file bar was transferred in ascii mode.

grep

Use this command to search for information in a file or files. For example, suppose that we have a file dict whose contents are

   red rojo

   green verde

   blue azul

   white blanco

   black negro

Then we can look up items in our file like this;

   % grep red dict

     red rojo

  % grep blanco dict

     white blanco

   % grep brown dict

Notice that no output was returned by grep brown. This is because "brown" is not in our dictionary file.
Grep can also be combined with other commands. For example, if one had a file of phone numbers named "ph", one entry per line, then the following command would give an alphabetical list of all persons whose name contains the string "Fred".

   % grep Fred ph | sort

     Alpha, Fred: 333-6565

     Beta, Freddie: 656-0099

     Frederickson, Molly: 444-0981

     Gamma, Fred-George: 111-7676

     Zeta, Frederick: 431-0987

The symbol "|" is called "pipe." It pipes the output of the grep command into the input of the sort command.

For more information on grep, consult

   % man grep

head

Use this command to look at the head of a file. For example,

   % head essay.001

displays the first 10 lines of the file essay.001 To see a specific number of lines, do this:

   % head -20 essay.001

This displays the first 20 lines of the file.

ls

Use ls to see what files you have. Your files are kept in something called a directory.

   % ls

     foo       letter2

     foobar    letter3

     letter1   maple-assignment1

Note that you have six files. There are some useful variants of the ls command:

   % ls l*

     letter1 letter2 letter3

Note what happened: all the files whose name begins with "l" are listed. The asterisk (*) is the " wildcard" character. It matches any string.

lpr

This is the standard Unix command for printing a file. It stands for the ancient "line printer." See

   % man lpr

for information on how it works. See print for information on our local intelligent print command.

Basic Unix Commands-----1

Unix Command Summary

See the Unix tutorial for a leisurely, self-paced introduction on how to use the commands listed below. For more documentation on a command, consult a good book, or use the man pages. For example, for more information on grep, use the command man grep.

cat --- for creating and displaying short files
chmod --- change permissions
cd --- change directory
cp --- for copying files
date --- display date
echo --- echo argument
ftp --- connect to a remote machine to download or upload files
grep --- search file
head --- display first part of file
ls --- see what files you have
lpr --- standard print command (see also print )
more --- use to read files
mkdir --- create directory
mv --- for moving and renaming files
ncftp --- especially good for downloading files via anonymous ftp.
print --- custom print command (see also lpr )
pwd --- find out what directory you are in
rm --- remove a file
rmdir --- remove directory
rsh --- remote shell
setenv --- set an environment variable
sort --- sort file
tail --- display last part of file
tar --- create an archive, add or extract files
telnet --- log in to another machine
wc --- count characters, words, lines

cat

This is one of the most flexible Unix commands. We can use to create, view and concatenate files. For our first example we create a three-item English-Spanish dictionary in a file called "dict."

   % cat >dict

     red rojo

     green verde

     blue azul

<control-D>

<control-D> stands for "hold the control key down, then tap 'd'". The symbol > tells the computer that what is typed is to be put into the file dict. To view a file we use cat in a different way:

   % cat dict

     red rojo

     green verde

     blue azul

If we wish to add text to an existing file we do this:

   % cat >>dict

     white blanco

     black negro

     <control-D>

Now suppose that we have another file tmp that looks like this:

   % cat tmp

     cat gato

     dog perro

Then we can join dict and tmp like this:

   % cat dict tmp >dict2

We could check the number of lines in the new file like this:

   % wc -l dict2

The command wc counts things --- the number of characters, words, and line in a file.

chmod

This command is used to change the permissions of a file or directory. For example to make a file essay.001 readable by everyone, we do this:

   % chmod a+r essay.001

To make a file, e.g., a shell script mycommand executable, we do this

   % chmod +x mycommand

Now we can run mycommand as a command.

To check the permissions of a file, use ls -l . For more information on chmod, use man chmod.

cd

Use cd to change directory. Use pwd to see what directory you are in.

   % cd english

   % pwd

   % /u/ma/jeremy/english

   % ls

novel poems

   % cd novel

   % pwd

   % /u/ma/jeremy/english/novel

   % ls

ch1 ch2 ch3 journal scrapbook

   % cd ..

   % pwd

   % /u/ma/jeremy/english

   % cd poems

   % cd

   % /u/ma/jeremy

Jeremy began in his home directory, then went to his english subdirectory. He listed this directory using ls , found that it contained two entries, both of which happen to be diretories. He cd'd to the diretory novel, and found that he had gotten only as far as chapter 3 in his writing. Then he used cd .. to jump back one level. If had wanted to jump back one level, then go to poems he could have said cd ../poems. Finally he used cd with no argument to jump back to his home directory.

cp

Use cp to copy files or directories.

   % cp foo foo.2

This makes a copy of the file foo.

   % cp ~/poems/jabber .

This copies the file jabber in the directory poems to the current directory. The symbol "." stands for the current directory. The symbol "~" stands for the home directory.

date

Use this command to check the date and time.

   % date

Fri Jan  6 08:52:42 MST 1995

Data warehousing concepts (FAQ’s)

Data warehousing concepts (FAQ’s)

1) What is Data warehouse?

Data warehouse is relational database used for query analysis and reporting. By definition data warehouse is Subject-oriented, Integrated, Non-volatile, Time variant.

Subject oriented : Data warehouse is maintained particular subject.

Integrated : Data collected from multiple sources integrated into a

user readable unique format.

Non volatile : Maintain Historical date.

Time variant : data display the weekly, monthly, yearly.

2) What is Data mart?

A subset of data warehouse is called Data mart.

3) Difference between Data warehouse and Data mart?

Data warehouse is maintaining the total organization of data. Multiple data marts used in data warehouse. where as data mart is maintained only particular subject.

4) Difference between OLTP and OLAP?

OLTP is Online Transaction Processing. This is maintained current transactional data. That means insert, update and delete must be fast.

5) Explain ODS?

Operational data store is a part of data warehouse. This is maintained only current transactional data. ODS is subject oriented, integrated, volatile, current data.

6) Difference between Power Center and Power Mart?

Power center receive all product functionality including ability to multiple register servers and metadata across the repository and partition data.

One repository multiple informatica servers. Power mart received all features except multiple register servers and partition data.

7) What is a staging area?

Staging area is a temporary storage area used for transaction, integrated and rather than transaction processing.

When ever your data put in data warehouse you need to clean and process your data.

8) Explain Additive, Semi-additive, Non-additive facts?

Additive fact: Additive Fact can be aggregated by simple arithmetical additions.

Semi-Additive fact: semi additive fact can be aggregated simple arithmetical

additions along with some other dimensions.

Non-additive fact: Non-additive fact can’t be added at all.

9) What is a Fact less Fact and example?

Fact table which has no measures.

10) Explain Surrogate Key?

Surrogate Key is a series of sequential numbers assigned to be a primary key for the table.

11) How many types of approaches in DHW?

Two approaches: Top-down(Inmol approach), Bottom-up(Ralph Kimball)

12) Explain Star Schema?

Star Schema consists of one or more fact table and one or more dimension tables that are related to foreign keys.
Dimension tables are De-normalized, Fact table-normalized

Advantages: Less database space & Simplify queries.

13) Explain Snowflake schema?

Snow flake schema is a normalize dimensions to eliminate the redundancy.The dimension data has been grouped into one large table. Both dimension and fact tables normalized.

14) What is confirm dimension?

If both data marts use same type of dimension that is called confirm dimension.If you have same type of dimension can be used in multiple fact that is called confirm dimension.

15) Explain the DWH architecture?

16) What is a slowly growing dimension?

Slowly growing dimensions are dimensional data,there dimensions increasing dimension data with out update existing dimensions.That means appending new data to existing dimensions.

17) What is a slowly changing dimension?

Slowly changing dimension are dimension data,these dimensions increasing dimensions data with update existing dimensions.

Type1: Rows containing changes to existing dimensional are update in the target by overwriting the existing dimension.In the Type1 Dimension mapping, all rows contain current dimension data.

Use the type1 dimension mapping to update a slowly changing dimension table when you do not need to keep any previous versions of dimensions in the table.

Type2: The Type2 Dimension data mapping inserts both new and changed dimensions into the target.Changes are tracked in the target table by versioning the primary key and creating a version number for each dimension in the table.

Use the Type2 Dimension/version data mapping to update a slowly changing dimension when you want to keep a full history of dimension data in the table.version numbers and versioned primary keys track the order of changes to each dimension.

Type3: The type 3 dimension mapping filters source rows based on user-defined comparisions and inserts only those found to be new dimensions to the target.Rows containing changes to existing dimensions are updated in the target. When updating an existing dimension the informatica server saves existing data in different columns of the same row and replaces the existing data with the updates.

18) When you use for dynamic cache.

Your target table is also look up table then you go for dynamic cache .In dynamic cache multiple matches return an error.use only = operator.

19) what is lookup override?

Override the default SQL statement.You can join multiple sources use lookup override.By default informatica server add the order by clause.

20) we can pass the null value in lookup transformation?

Lookup transformation returns the null value or equal to null value.

21) what is the target load order?

You specify the target load order based on source qualifiers in a mapping.if u have the multiple source qualifiers connected to the multiple targets you can designate the order in which informatica server loads data into the targets.

22) what is default join that source qualifier provides?

Inner equi join.

23) what are the difference between joiner transformation and source qualifier transformation?

You can join heterogeneous data sources in joiner transformation, which we cannot achive in source qualifier transformation.

You need matching keys to join two relational sources in source qualifier transformation.where you doesn’t need matching keys to join two sources.

Two relational sources should come from same data source in source qualifier.You can join relational sources, which are coming from different sources in source qualifier.You can join relational sources which are coming from different sources also.

24) what is update strategy transformation?

Whenever you create the target table whether you are store the historical data or current transaction data in to target table.

25) Describe two levels in which update strategy transformation sets?

26) what is default source option for update strategy transformation?

Data driven.

27) What is data driven?

The information server follows instructions coded into update strategy transformations with in the session mapping determine how to flag records for insert,update,delete or reject if u do not choose data driven option setting , the informatica server ignores all update strategy transformations in the mapping.

28) what are the options in the trarget session of update strategy transformation?

Insert

Delete

Update

Update as update

Update as insert

Update else insert

Truncate table.

29) Difference between the source filter and filter?

Source filter is filtering the data only relational sources. Where as filter transformation filter the data any type of source.

30) what is a tracing level?

Amount of information sent to log file.

-- What are the types of tracing levels?

Normal,Terse,verbose data,verbose intitialization.

--Expalin sequence generator transformation?

-- can you connect multiple ports from one group to multiple transformations?

Yes

31) can you connect more than one group to the same target or transformation?

32) what is a reusable transformation?

Reusable transformation can be a single transformation.This transformation can be used in multiple mappings.when you need to incorporate this transformation into mapping you add an instance of it to mapping.Later if you change the definition of the transformation, all instances of it inherit the changes.Since the instance of reusable transformation is a pointer to that transformation.U can change the transformation in the transformation developer, its instance automatically reflect these changes. This feature can save U great deal of work.

-- what are the methods for creating reusable transformation?

Two methods

1) Design it in the transformation developer.

2) Promote a standard transformation from the mapping designer.After you add a transformation to the mapping, you can promote it to status of reusable transformation.

Once you promote a standard transformation to reusable status, you can demote it to a standard transformation at any time.

If u change the properties of a reusable transformation in mapping , you can revert it to the original reusable transformation properties by clicking the revert.

33) what are mapping parameters and mapping variables?

Mapping parameter represents a constant value that you can define before running a session.A mapping parameter retains the same value throughout the entire session.

When you use the mapping parameter , you declare and use the parameter in a mapping or mapplet.Then define the value of parameter in a parameter file for the session.

Unlike a mapping parameter, a mapping variable represents a value that can change through out the session. The informatica server save the value of mapping variable to the repository at the end of session run and uses that value next time you run the session.

34) can you use the mapping parameters or variables created in one mapping into another mapping?

NO, we can use mapping parameters or variables in any transformation of the same mapping or mapplet in which have crated mapping parameters or variables.

35) Can you are the mapping parameters or variables created in one mapping into any other result transformation.

Yes because the reusable transformation is not contained with any mapplet or mapping.

36) How the informatica server sorts the string values in rank transformation?

When the informatica server runs in the ASCII data movement mode it sorts session data using binary sort order.If you configures the session to use a binary sort order, the informatica server calculates the binary value of each string and returns the specified number of rows with the highest binary values for the string.

37) What is the rank index in rank transformation?

The designer automatically creates a RANKINDEX port for each Rank transformation. The informatica server uses the Rank Index port to store the ranking position for each record in a group.For example, if you create a Rank transformation that ranks the top 5 sales persons for each quarter, the rank index number the salespeople from 1 to 5.

38) what is the mapplet?

Mapplet is a set of transformation that you build in the mapplet designer and you can use in multiple mappings.

39) Difference between mapplet and reusable transformation?

Reusable transformation can be a single transformation.Where as mapplet use multiple transformations.

40) what is a parameter a file?

Paramater file defines the values for parameter and variables.

WORKFLOW MANAGER

41) what is a server?

The power center server moves data from source to targets based on a workflow and mapping metadata stored in a repository.

42) what is a work flow?

A workflow is a set of instructions that describe how and when to run tasks related to extracting,transformation and loading data.

-- what is session?

A session is a set of instructions that describes how to move data from source to target using a mapping.

-- what is workflow monitor?

Use the work flow monitor work flows and stop the power center server.

43) explain a work flow process?

The power center server uses both process memory and system shared memory to perform these tasks.

Load manager process: stores and locks the workflow tasks and start the DTM run the sessions.

Data Transformation Process DTM: Perform session validations,create threads to initialize the session,read,write and transform data, and handle pre and post session operations.

The default memory allocation is 12,000,000 bytes.

44) What are types of threads in DTM?

The main dtm thread is called the master thread.

Mapping thread.

Transformation thread.

Reader thread.

Writer thread.

Pre-and-post session thread.

45) Explain work flow manager tools?

1) Task developer.

2) Work flow designer.

3) Worklet designer.

46) Explain work flow schedule.

You can sehedule a work flow to run continuously, repeat at given time or interval or you manually start a work flow.By default the workflow runs on demand.

47) Explain stopping or aborting a session task?

If the power center is executing a session task when you issue the stop the command the power center stop reading data. If continuous processing and writing data and committing data to targets.

If the power center can’t finish processing and committing data you issue the abort command.

You can also abort a session by using the Abort() function in the mapping logic.

48) What is a worklet?

A worklet is an object that represents a set of taske.It can contain any task available in the work flow manager. You can run worklets inside a workflow. You can also nest a worklet in another worklet.The worklet manager does not provide a parameter file for worklets.

The power center server writes information about worklet execution in the workflow log.

49) what is a commit interval and explain the types?

A commit interval is the interval at which power center server commits data to targets during a session. The commit interval the number of rows you want to use as a basis for the commit point.

Target Based commit: The power center server commits data based on the number of target rows and the key constraints on the target table. The commit point also depends on the buffer block size and the commit interval.

Source-based commit:---------------------------------------------

User-defined commit:----------------------------------------------

50) Explain bulk loading?

You can use bulk loading to improve performance of a session that inserts a large amount of data to a db2,sysbase,oracle or MS SQL server database.

When bulk loading the power center server by passes the database log,which speeds performance.

With out writing to the database log, however the target database can’t perform rollback.As a result you may not be perform recovery.

51) What is a constraint based loading?

Introduction to DWH

A Data Warehouse -Ralph Kimball [father of DWH-ii]

1. A Data warehouse is a relational database management system which is specifically designed for business analysis purpose, rather than for Business Transactional Processing.

2. A data Ware house is designed to support Decision Making Processing [DMP].

3. Decision Support System [DSS]:- since that Data Warehouse is designed to support Decision Making Processing. Hence it is call as DSS.

Read only Database

The data warehouse is designed to make query the data required for analysis but not for business Transactional processing. Hence it is known as Read only Database.

Historical Database

Since a data warehouse contains a historical data which is required for analysis. Hence it is known Historical Database.

A Data Warehouse-W.H. Inmon[father of DWH-I]

A data warehouse is a 1. Time variant, 2.Non-volatile, 3.Subject Oriented, 4. Integrated database.

Characteristics features of Data Warehouse

1. Time Variant: - A data warehouse is time variant database which support the business users in analysis the business and companies the business with different time periods. This is known as time series analysis.

Data warehouse

2. Non-volatile:- A data warehouse is non-volatile database, once the data enter into the data warehouse it does not reflex to the change which takes place at operational source system. Hence the data is static in Data warehouse.

3. Integrated Database: - A data warehouse is an integrated database which store data in integrated format collected from multiple operational source systems.

4. Subject oriented:-A Data warehouse is subject oriented database which support the business needs of middle management users. A subject represents department specification business needs.

Example: HR, Sales, Accounts, Marketing, Loans, Insurance.

Differences between Operational Database and Data warehouse.

Operational Database Data warehouse

1. It is designed to support 1. It is designed to support Decision Making Decision making Processing. processing

2. Designed for Business Operation. 2. Decision for Business analysis.

3. Volatile data 3. Non-volatile data.

4. Current data 4. Historical data.

5. Less history [3-6 months] 5. More history [15-20 years].

6. Detail data 6. Summary data.

7. Normalization 7. De-normalization.

8. Designed for clerical access 8. Managerial access.

9. E-R Modeling 9. Dimensional Modeling.

10. More joins 10.Few joins.

Data Marts and Types of Data Marts

A data Marts is subset of an Enterprise Data warehouse. A data mart is subject
oriented database which supports business needs of middle management
or department specific users. A data mart is also known as High Performance
Query Structures (HPQS).

Data warehouse store enterprises specific historical data.

There are 2 types’ data marts.

1. Dependent data marts.

2. Independent data marts.

Difference between data mart and enterprise data warehouse

Data Mart Data warehouse

1. It is a department specific database it is an enterprises specific database.

2. Sigel subject integration of multiple subjects.

3. Middle management access top management access.

Top down data warehousing approach [W.H.INMON]

According to the w.h.inmon first we need to design enterprises specific database
known as data warehouse, from the enterprise data warehouse derive the
subject oriented database known as data marts.

Dependent data mart

In a top down approach a data mart development depends on an enterprise data
warehouse. Hence such data mart known as dependent data marts.

Bottom up approach [Ralf Kimball]

According to the Kimball first designed the department specific databases known as
data marts integrated the data marts into an enterprise data warehouse.

Independent data marts

In the bottom up approach data mart development is independent of an enterprise
data warehouse such data marts known as independent data marts.

Operational Data Store[ODS 60-90days]

An ODS is integrated view of operational data sources or operational databases
designed to support business transactional processing.

Differences between ODS and DWH

1. Integrated database integrated database.

2. Subject oriented subject oriented.

3. Detail data summary data.

4. Current data[60-90 days] historical data[15-20 years]

5. Volatile data non volatile data.

6. Normalization de normalization.

Data Warehouse and Informatica Realtime Scenarios