1 Before You Start: Binary and Text Files

1.1 Variations among Text Files

1.1.1 ASCII

1.1.2 ASCII Line Termination

Even if you have ASCII text in a file, there is some variation on how to encode the file. Windows and other operating systems disagree on how to divide an ASCII text file into lines.

• The end of a line in a Unix/Linux/Android/OSX ASCII text file is represented by a single character: the LF (line feed) character, a.k.a. ^J.

• MS Windows uses a pair of characters at the end of each line: the CR (carriage return) or ^M followed by a LF.

This means that ASCII text files created in Windows tend to look, to a Unix program, as if they have extra ^M characters near the end of each line. ASCII files created in Unix, on the other hand, tend to look to a Windows program as if the entire file contains only one line with odd ^J characters sprinkled inside.

Example 1: Try This:

1. Log into one of our Linux servers.

   Do

   more /home/cs252/Assignments/textFiles/wintext.txt 
   more /home/cs252/Assignments/textFiles/linuxtext.txt 
   more /home/cs252/Assignments/textFiles/mixedtext.txt
   

   You won’t see a difference, because more is smart enough to handle both styles of line termination.

2. Give the command

   diff /home/cs252/Assignments/textFiles/wintext.txt /home/cs252/Assignments/textFiles/linuxtext.txt 
   

   Remember that diff compares two files and lists the liens that are different. In this case, it list all of the lines in each file.

3. Give the command

   diff -b /home/cs252/Assignments/textFiles/wintext.txt /home/cs252/Assignments/textFiles/linuxtext.txt 
   

   The -b option tells diff to ignore blanks and other “whitespace”, i.e., characters that take up space but don’t produce any visible “ink” in their place. Whitepace characters include blanks, tab characters, and carriage returns.

   When ignoring carriage returns, diff considers the files to be identical. It doesn’t list any liens at all.

4. Give the commands

   file /home/cs252/Assignments/textFiles/wintext.txt 
   file /home/cs252/Assignments/textFiles/linuxtext.txt 
   file /home/cs252/Assignments/textFiles/mixedtext.txt
   

   Observe how the file command tells you about the contents of each.

1.2 Identifying the File Contents

How can you tell if a file is text or binary?

Let’s get one thing out of the way right now:

You can not tell if a text is text or binary by double-clicking on it in an operating system window to open it up. Launching a file in this way simply runs whatever program the operating system believes is most appropriate to that file. That program may very well show you text, but that doesn’t mean the information was encoded as a text file. On the other hand, the program might show you graphics with no text at all, but that does not mean that the graphics were not drawn from a description written in ASCII text.

You can get a hint as to whether a file is text or binary by looking at the file extension (the 2 or 3 letters after the final ‘.’ in a file name).

• These are usually text files:

  • *.txt and anything else that you produce with a text editor (but not a word processor)

  • Source code: *.h, *.cpp, *.java, makefile

  • Web pages and web-based data: *.html, *.htm, *.xml, *.xhtml

  • A few graphics formats: *.ps, *.eps, *.svg

• These are almost always binary files:

  • Things produced by a compiler: *.o, *.class, *.a, *.lib, *.dll, *.exe

  • Formatted documents other than web pages: *.doc, *.docx, *.pdf

  • Most graphics formats: *.gif, *.jpeg, *.jpg, *.png, *.tiff, *.bmp

    • But not *.svg

  • All music and video formats

  • Compressed files and archives: *.zip, *.tar, *.jar, *.7z, *.rar, *.gz, *.Z

• These could be either

  • *.dat

  • Files whose names do not include an extension at the end

But that’s only a hint. There are so many programs in the world that some are bound to use the same extensions.

In Linux, the best way to see what kind of data is in a file is to use the file command, e.g.,

file mystery.dat

The file command will print a description fo the file contents. This description can be a few lines long. If the file contains ASCII text or Unicode text, this will be stated explicitly as part of the description. If the file is ASCII text but with Windows-style line termination, it will state that as well.

In Windows, your best way to see if a file is text or binary is to open it in a text editor such as NotePad (but not in a word processor such as Word).

• If the result is readable text, you have ASCII text with Windows-style line terminators.

• If the result is readable text but with everything crammed into one line, you have ASCII text with *nix-style line terminators.

• If the result has lots of white or black rectangles in the place of characters, the file is apparently binary.

1.3 Converting Text Files

If you find yourself with a Windows-style text file on a Linux machine, or if you have a Linux-style text file that you want to transfer to a Windows machine, you can convert from one form to the other.

Important! Make sure that the file you are working on really is text before trying any of these conversions. If you use any of these on a binary file, you will corrupt the file past the point of possible repair.

That’s precisely why we spent so much time, above, explaining how to tell if a file is binary or text.

1.3.1 Windows to Linux

To convert a Windows-style text file to Linux, use the command tr:

tr -d '\r' < file₁ > file₂

1.3.2 Linux to Windows

You can also prepare a text file for transfer to a Windows system with unix2dos:

unix2dos file₁

Be sure to check your file with the file command before using dos2unix or unix2dos. If the file is not, in truth, ASCII text, these commands will likely leave you with a badly corrupted file.

2 Transferring Files Across the ODU Local Network: Samba

If you are sitting at a Windows PC that is part of either the CS Dept’s own local network (e.g., in the CS Dept. labs or connected to the CS Dept’s wireless network) or part of the ODU ITS local network (most ODU computer labs on the Norfolk, Virginia Beach, and Peninsula Gradate Center campuses), then you can access your Unix account directories directly from within Windows. This is because the CS Dept. Unix file servers run a service called “Samba”, a program that mediates file access between UNIX and other systems.

Again, let me emphasize that Samba only works on a local network. If you are connecting to the campus via the Internet, forget Samba.

To use Samba, you might not need to do anything at all. If you are logged in to a CS Dept PC and you have a Z: drive mapped, that is actually a Samba connection to your Unix home directory.

If you have no such drive, use the Windows “Start->Run” button to run

\\userdata.cs.odu.edu\undergrad\your-login-name

(Graduate students would use “grad” instead of “undergrad” in the path above.) You may be prompted for a password, or a login name and password. Supply your CS Unix login/password and, if all is well, a Windows Explorer window should open displaying the contents of your Unix home directory. You can now manipulate files in this Window just as you would in any Windows directory/folder, but the changes are occurring in your Unix directory.

Now that we know that it works, we can make this whole process more convenient by mapping a Windows drive letter to your Unix account, giving you a “fake” disk drive that actually accesses your Unix files. From inside any Windows Explorer or “ My Computer” window, select “Tools” (or right-click on the “My Computer” icon) and select “ Map Network Drive…”. Select an unused drive letter, and enter that same address/command string as in the last step for the “ Folder”. Make sure the “Reconnect at logon” box is checked. Finally, if your login name for logging into the PC is different from your CS Unix login name, look for a “Connect using a different user name” link, click on that and supply your Unix login information. Click on OK/Finish and within a few seconds, you should have a new drive available that actually maps onto your Unix account.

Two things to keep in mind when using Samba to access your Unix files from Windows:

3 Transferring Files Across the Internet: ftp

FTP (File Transfer Protocol) is the mechanism for transferring files over the Internet. Although most browsers provide some support for FTP, they usually only permit downloads (from the remote machine to your local PC) and usually only permit access to public repositories, not to password-protected accounts.

FTP (File Transfer Protocol) is one of the oldest mechanisms for for transferring files over the Internet. Although most browsers provide some support for FTP, they usually only permit downloads (from the remote machine to your local PC) and usually only permit access to public repositories, not to password-protected accounts.

From a modern perspective, FTP has a number of limitations.

• It has no encryption. Anyone on the network could, theoretically, see what you are transferring.

• Even worse, if you are transferring to a machine that requires you to log in, FTP transmits your login name and password in plain-text, unencrypted form.

• FTP struggles with modern firewalls and routers.

Consequently, FTP has been largely supplanted in practice by SFTP (the Secure File Transfer Protocol) which leverages the built-in encryption of SSH and a special ability of SSH to set up secure, router & fire-wall friendly, “tunnels” through which other protocols can work.

Still, you may occasionally see a download link from a public (no-login) repository with an ftp://… URL. And many programs that support SFTP will also offer an FTP option that, in most cases, you should simply ignore.

4 Secure File Transfer: sftp and scp

SFTP (Secure File Transfer Protocol) is a more modern file transfer that encodes your entire session. It is built upon the secure ssh service, and therefore shares ssh’s ability to tunnel out through most reasonably configured firewalls.

Most SSH servers are also SFTP servers, so if you have an account on a machine that lets you use an SSH client program to open command sessions, you can probably use an SFTP client program to open file transfer sessions on that same server.

You will need an SFTP client program on your local machine to do this.

• Many systems have sftp commands that can be used to transfer files via a text-based interface.

• Alternatively, some programs provide GUI-based interfaces to SFTP that allow you to initiate file transfers by a simple “drag and drop”.

We’ll look at both styles of SFTP client.

4.1 SFTP via a Text-based Interface

Text-based sftp clients are widely available. Linux and OS/X machines will have one as part of the standard OS distribution. Windows does not come with one, but if you have isntalled the Windows 10 OpenSSH, that will have included a text-based sftp command.

Typically, a text-based client is launched from your local machine with the command

sftp yourLoginName@serverName

The one potentially confusing thing about working with the sftp command is that you need to be constantly aware of what directory you are working with on two different machines: the local machine you are issuing the commands from and the remote machine that you have connected to.

The commands provided by a typical text-based SFTP client are:

• pwd

  Prints the current directory that you ware working in on the remote machine.

  • You can print your current working directory on the local machine by putting an ‘l’ on the front of the command: lpwd

• cd directory

  Changes to a different directory on the remote machine.

  • You can change directories on your local machine by putting an ‘l’ on the front of the command: cd directory

    Tip: If you know that you are going to transferring files from one specific directory on your local PC, it’s easiest to cd to that directory before you even run the sftp command. That way you won’t need to do any lcd commands because you will already be in the right place on your local machine.

• ls

  Lists the contents of the current working directory on the remote machine.

  • Again, you can list directories on your local machine by putting an ‘l’ on the front of the command: lls

• get filename

  Download a file from the remote machine onto the local machine.

  • Most SFTP clients will allow wildcard patterns so that you can easily download multiple files at once.

• put filename

  Upload a file from your local machine to the remote machine.

  • Most SFTP clients will allow wildcard patterns so that you can easily upload multiple files at once.

• quit

  Just what it says. exit also works.

Here is a sample SFTP session:

The commands supplied by the user are shown like this.

sftp yourName@linux.cs.odu.edu  ➀ 
Connected to linux.cs.odu.edu. 
yourName@linux.cs.odu.edu's password: 
sftp> lcd courseData/temp                ➁
sftp> cd data 
sftp>  ls
file1.txt  file2.dat  file3.txt  file4.dat   ➂
sftp>  get file1.txt               ➃
Fetching /home/yourName/data/file1.txt to file1.txt 
/home/yourName/data/file1.txt                     100%   20KB  19.6KB/s   00:0 226 Transfer complete 
sftp>  get *.dat                   ➄
Fetching /home/yourName/data/file2.dat to file2.dat 
/home/yourName/data/file2.dat                     100% 1001 1.0KB/s   00:
Fetching /home/yourName/data/file4.dat to file4.dat 
/home/yourName/data/file4.dat 100% 2123     2.1KB/s   0 226 Transfer complete 
sftp> quit 

• ➀ You would, of course, need to use your own login name instead of “yourName”

• ➁ Here we change to the directory on our local machine that contains the data we want to send or the location that we want to receive data.

  You can avoid the need to do most lcd commands by cd’ing to the relevant directory before issuing the sftp command, so that you are already in the correct working directory when SFTP starts up.

• ➁ Note that these are files on the remote server machine.

• ➂ Downloading one file.

• ➃ Using wildcards to download multiple files.

4.2 SFTP via GUI Interface

4.3 Try It Out

5 Fetching Files from Across the Internet: wget

Finally, there is a convenient way to get a copy of a file that is provided via a web server. The wget command accepts a URL (a web address), fetches the file at that URL, and deposits a copy of that file in your current directory.