This is a list of evaluation criteria that have been applied by the authors of censorship circumvention systems and papers.

Many of these criteria deal with a specific threat model; not all of them are appropriate for every circumvention system. No one evaluation applied more than a fraction of the criteria. We tried to be comprehensive in our survey, and do not necessarily think that every criterion we found is good or necessary.

Questions? Contact the authors of the survey:

Revised 2017-07-28.


GOAL: adaptability to blocking

Assesses the system's resilience to unexpected blocking events and new kinds of blocking.

time to create an adaptation

The amount of time it takes some programmer to create a new adaptation of the protocol.

GOAL: availability of infrastructure

This criterion considers the availability of the infrastructure used by an approach for users and the feasibility of deployment.

geographic diversity of proxies

Examines the diversity of proxies in terms of geographic location.

independent deployment

Can deploy the circumvention approach without needing the help of third-parties, such as friendly ISPs.

number of proxies

The number of proxies usable with the tool.

preponderance of suitable servers

Some tools rely on finding servers with a specific property at large in the Internet. For example, servers that accept user-posted content, or servers with specific network protocol quirks. This criterion means that the study measured how common such servers really are.

rate of proxy churn

Measures or estimates the rate at which new proxies appear and old proxies go away.

stability of decoy hosts

Examines how long a decoy host is available to carry on a conversation.

GOAL: client performance

Assesses an approach's client program efficiency in terms of CPU and memory usage.

CPU usage by users

Assesses the percentage of CPU cycles consumed by the client or server part of the system.

memory usage by users

Assesses the memory requirements to run the system.

startup time

Measures how quickly client software starts up.

GOAL: decentralization of trust

Evaluates the degree to which trust is centralized; i.e., all trust is placed in a single server/company, or spread out among many parties.

GOAL: deniability under computer inspection

Whether the circumvention tool users can plausibly deny using it even when their computers are inspected.

no installation

Using the tool does not require installing special software.

GOAL: developed by experts

Assesses whether the developers of the tool are known security experts.

GOAL: diversity of users

Assesses the diversity of types of users of the system.

GOAL: end-user protection

Assesses whether an approach protects user's privacy from intermediate and end nodes.

does not store user information

Does not log user information.

hide user information from end host

The tool hides information about the user from the end host (destination) to which the user connects to provide some degree of anonymity.

GOAL: infrastructure cost

Assesses the cost of infrastructure required to deploy an approach in real world.

cost of external services

Estimates the cost of external services, e.g., cloud services, CDNs, that are required to deploy an approach.

GOAL: network performance

Assesses the system's performance in terms of goodput, latency and overhead.

byte overhead

How many extra bytes are introduced by the tool.

fraction of clients that can utilize the network

Assesses the number of clients (source hosts) in the Internet that would be able to join a system.

goodput

The amount of useful throughput the tool enables.

latency

Assesses the round-trip time for a request.

number of requests needed to retrieve data

Assesses the number of requests that a requester must make to retrieve hidden messages.

registration performance

Some systems need to apply a special distinguisher or mark to traffic destined for circumvention. For example, end-to-middle proxying systems need to tag flows at the client side and recognize them at the station. This criterion considers the performance of the registration method.

speed of downloading a webpage

Assesses the time required to download a webpage. This is really a combination of goodput and latency, but it is specifically applied so often that we made it its own criterion.

throughput

The amount of throughput/bandwidth the tool enables.

time overhead

How much extra time it takes to use the tool.

GOAL: openness of design

Assesses whether the source code available (client and server) and whether the design public or relies on security through obscurity.

open source

The tool's source code is open.

GOAL: resistance to active probing

Active probing attacks involve the censor initiating connections to hosts to determine whether the host runs a given circumvention protocol, typically then blocking the host's IP address upon finding that it does. A system is resistant to active probing if an adversary cannot discover the use of the system using this technique.

respond to probes like something else

When probed, respond similar to how some allowed server would respond so that a censor deciding to block such responses will incur false positives.

use authentication

Whether a client needs authentication to connect to the server.

GOAL: resistance to blocking

A system resists blocking if it is hard to block the protocol or IP address of the infrastructure that the approach uses, even given a method of identifying it. For example, if blocking would cause substantial collateral damage.

use a popular protocol

Whether an approach sends traffic using a popular protocol, such as the Skype protocol, to force the censor to either block a popular protocol or identify the circumventing usage of the protocol from normal usage.

use many access points

Whether an approach uses too many hosts to make it hard for a censor to block all of them.

use network infrastructure

Whether an approach uses infrastructure within a network, e.g., router, to avoid address blocking.

use popular hosts

Whether an approach uses popular hosts, such as Skype nodes and CDNs, to resist address blocking.

GOAL: resistance to insider attacks

Considers whether the system continues to work even if the censor joins the circumvention network and attempts to disrupt it.

GOAL: resistance to security attacks

This criterion considers different measures that a paper uses to avoid security attacks such as man-in-the-middle, denial of service, malicious proxy, key reuse and replay attack.

ignore invalid connections (DoS)

Whether an approach ignores invalid connections to avoid denial of service attacks.

limit service to each user ID (DoS)

Avoid DoS by limiting the amount of service the approach will provide to each user.

use TLS for confidentiality

Whether an approach uses TLS to provide confidentiality.

use authenticated key exchange (MITM)

Whether an approach uses authenticated key exchange.

use block cipher (key reuse)

Whether an approach uses block cipher to resist key reuse attack.

use certificate pinning (MITM)

Whether an approach uses certificate pinning to avoid MiTM.

use client puzzle (DoS)

Require clients to solve a puzzle to prevent DoS.

use encryption for confidentiality

Whether an approach uses encryption for confidentiality (and/or integrity).

use shared secret (MITM)

Whether an approach uses shared secret to resist man-in-the-middle attacks.

use strong third-party service (DoS)

A censor would have to overcome not just the circumvention deployment, but some third-party that hosts the deployment.

use timestamp (replay)

Whether an approach uses timestamp to resist replay attack.

use trustworthy proxy

By using a trustworthy proxy as the forwarder, the approach avoids the risks of a malicious proxy (as long as the proxy remains trustworthy).

GOAL: resistance to traffic analysis

An approach is resistant to traffic analysis if an adversary cannot statically use properties of the traffic generated by the approach to detect it. (Some of the metrics used for this goal can also be used for active probing, but they are not inherently active.)

TLS characteristics

Prevents detection by TLS characteristics, like TLS nonce, clienthello or serverhello messages.

concurrent connection count

They measured the number of simultaneous connections to a server.

connection length

Measures the duration of flows (e.g., TCP connections) and evaluates whether the duration can be a distinguisher. (If a connection is suspiciously long, for example.)

inter-packet timing

Measures the distribution of packet timing (interpacket times or packet rates) to assess whether it is unlike that of a blocked protocol, or like that of an allowed protocol.

lack of server response

This criterion is specific to TapDance and the way it sends half-finished HTTP requests. If a TapDance station exists along a path, the request is intercepted and a response comes back immediately. If there is no TapDance station, then no response ever comes (or perhaps comes only after a long timeout).

matching allowed n-gram distribution

Considers the distribution of consecutive strings of symbols, for example bytes. This includes 1-grams (e.g., distribution of single byte values).

network stack fingerprinting

Assesses the fingerprintability of the network stack, for example comparing the TCP options of two different hosts. Relevant when one host spoofs packets on behalf of another.

number of HTTP requests/responses

Measures the total number of HTTP request-response pairs per TCP connection.

packet size distribution

Measures the distribution of packet lengths to assess wehether it is unlike that of a blocked protocol, or like that of an allowed protocol.

protocol misclassification rate

Assesses the misclassification rate of the protocol classifiers to see how well the tool can evade the classifiers.

serial connection count

Counted the number of connections made in a row to a server.

total TCP connection

The total number of TCP connections per session does not stick out.

total payload length

The total payload length produced by the tool does not stick out.

use encryption to resist traffic analysis

Whether an approach uses encryption to resist traffic analysis.

use random port

Use a random port number for communications.

GOAL: resistance to traffic manipulation

Evaluates the system's resistance to modification of packets, or injecting or dropping packets. This criterion is concerned only with manipulation of client-initiated flows.

limit service to each user ID (DoS)

Avoid DoS by limiting the amount of service the approach will provide to each user.

use TLS for integrity

Whether an approach uses TLS to provide integrity.

use UDP with reliability

Whether an approach uses UDP with reliability.

use authentication

Whether a client needs authentication to connect to the server.

use error correcting codes

Whether an approach uses error correcting codes.

GOAL: self promotion

Evaluates whether an approach or tool promotes itself in a way that is likely to attract harmful attention (from the media or from the censor, for example).

GOAL: server obfuscation

Keeping the server used as a forwarder by the circumvention network hidden from the censor.

indirect connection to forwarder

The circumventor's computer connects indirectly to the circumvention network's forwarders through an innocuous server.

GOAL: sustainable network and development

Whether the system has funds and other resources to continue operating for the long term.

GOAL: usability

Assesses the additional effort that the circumvention tool client user must expand to use the system.

application support

Assesses the system's usefulness for a wide variety of applications (e.g. web browsing, chat, email).

availability of documentation

Assesses the quality of user and developer documentation.

clean uninstall

Assesses whether the client software leaves traces after being uninstalled.

free/low cost

Cost in USD to use the system.

has a GUI

Assesses whether the client software has a graphical user interface.

localization

Assesses whether the software and documentation are localized to relevant languages.

no installation

Using the tool does not require installing special software.

no usage limitation

Assesses whether an approach artificially limits who can use it and for which service.

number of errors per webpage

This criterion is specific to link-rewriting web proxies like CGIProxy that actually have to interpret HTML and JavaScript and change links so they point back into the proxy and not to their original location.

portability

Assesses the system's portability to different operating systems and devices.

small download file

The size of the tool's client program file is small.

software updates

Assesses the availability of software updates.

GOAL: usage

Assesses real world usage of an approach.

number of unique connections

Discusses the number unique of IP addresses that connect to the system on a daily basis.

number of users

The number of users the tool has.

test deployment

An approach proposed in an academic paper that is deployed in the real world and used by users.

GOAL: veracity of claims

Evaluates whether the claims of about an approach by the its provider match reality.