Web Analysis - Program Flow

Thomas J. Kennedy

Contents:

1 Program Flow

1.1 Data Analysis and Extraction

1.2 Further Analysis

1.3 Report Generation

1.4 Generating the Reports - Implementation

1 Program Flow

In total… there are three distinct phases into which WebAnalysis program logic can be divided:

Data Analysis and Extraction
Data Normalization (e.g., path and URL normalization)
Report Generation

1.1 Data Analysis and Extraction

Let us examine the actual analysis. Note that we will focus on the broad strokes. Details such as Exceptions, page count checks, path validation/checks, and URL validation will not be included.

PlantUML Markup

@startuml sequence_01.svg
autoactivate on
hide footbox
skinparam backgroundColor #FFFFFF
skinparam sequenceParticipant underline

title Data Analysis and Extraction: High-Level View

participant ":WebAnalysis" as driver
activate driver

create ":WebsiteBuilder" as wb

driver -> wb: new()
return

driver -> wb: withPath(path)
return

driver -> wb: withURLs(urls)
return

driver -> wb: build()
    wb -> wb: walkDirectory()
    return files
    wb -> wb: removeNonHTMLFiles(files)
    return prunedFiles

    create "site: Website" as site
    wb -> site: new(path, URLs)
    return

    loop for htmlFile in prunedFiles
        create ":BufferedReader" as buffer
        wb -> buffer: new(htmlFile)
        return
        create ":HTMLDocumentBuilder" as hb
        wb -> hb: new()
        return
        wb -> hb: withContentFrom(buffer)
        return
        wb -> hb: extractContent()
            hb -> hb: extractAnchors()
            return
            hb -> hb: extractImages()
            return
            hb -> hb: extractScripts()
            return
            hb -> hb: extractStyleSheets()
            return
        return
        wb -> hb: build()
            create "doc:HTMLDocument" as doc
            hb -> doc: new() [creation details not listed]
            return
        return doc 

        wb -> buffer !!: delete
        wb -> hb !!: delete

        wb -> site:addPage(doc)
        return
    end

return site

driver -> wb !!: delete

@enduml

Note that we did not cover the creation details of HTMLDocument objects within the HTMLDocumentBuilder.build method. I will leave this part up to you. Take note of how

all HTMLDocumentBuilder objects are destroyed
WebsiteBuilder is destroyed
all BufferedReader objects are destroyed

We have populated the site (Website object) with all the HTMLDocument objects. The analysis objects are no longer needed.

1.1.1 Creation & Deletion

Take note of the new and delete function “calls.”

new represents a call to a constructor (i.e., creating a new object)
delete represents an object being “deallocated” (i.e., leaving scope and being discarded)

Even though Java is a Garbage Collected language wherein memory management is not handled explicitly by a programmer… designing around (and making use of) object lifetimes is still important. (Even if some teachers are cavalier in their approach.)

1.2 Further Analysis

I will leave the intermediary steps (i.e., all steps until report generation) up to you and your team.

1.3 Report Generation

Since the Builder objects have been destroyed… we no longer need to consider them.

PlantUML Markup

@startuml sequence_03.svg
autoactivate on
hide footbox
skinparam backgroundColor #FFFFFF
skinparam sequenceParticipant underline

title Report Generation: High-Level View

participant "site:Website" as site
participant ":WebAnalysis" as driver
activate driver


create ":ReportManager" as manager

driver -> manager: new()
return
driver -> manager: setSourceData(site)
return

driver -> manager: determineBaseFileName()
return

driver -> manager: writeAll()
    create ":TextReportWriter" as textWriter
    manager -> textWriter: new()
    return
    manager -> textWriter: setSourceData(site)
    return
    manager -> textWriter: setBaseName(...)
    return
    manager -> textWriter: write()
    return
    manager -> textWriter !!: delete

    create ":JSONReportWriter" as jsonWriter
    manager -> jsonWriter: new()
    return
    manager -> jsonWriter: setSourceData(site)
    return
    manager -> jsonWriter: setBaseName(...)
    return
    manager -> jsonWriter: write()
    return
    manager -> jsonWriter !!: delete

    create ":ExcelReportWriter" as excelWriter
    manager -> excelWriter: new()
    return
    manager -> excelWriter: setSourceData(site)
    return
    manager -> excelWriter: setBaseName(...)
    return
    manager -> excelWriter: write()
    return
    manager -> excelWriter !!: delete
return

driver -> manager: writeReportNames(...)
return

@enduml

1.4 Generating the Reports - Implementation

Working with text files can be accomplished with the Java BufferedWriter and FileWriter classes. The other reports require external libraries. In total (including HTML parsing) the following libraries should be utilized…

Organization	Artifact ID	Version	Gradle Dependency	Description
org.jsoup	jsoup	1.16.1	org.jsoup:jsoup:1.16.1	HTML Parsing
org.apache.poi	poi	5.2.3+	org.apache.poi:poi:5.2.3+	Read/write Microsoft document formats
org.apache.poi	poi-ooxml	5.2.3+	org.apache.poi:poi-ooxml:5.2.3+	Read & write Excel spreadsheets
com.cedarsoftware	json-io	4.14.0	com.cedarsoftware:json-io:4.14.0	JSON Export/Import

The table outlines the libraries that students used in CS 350 during the Summer 2023 semester.

Note that Gradle is similar to make in that it handles compiling code. It also handles configuration management, generating documentation, running analysis tools…