|
|
|
Experimental Test Data (ICIQ 2012) |
| |
|
This page contains all test databases that we have used in the experiments of our research paper entitled "Key-based Blocking of Duplicates in Entity-Independent Probabilistic Data" that is published in the Proceedings of the 17th International Conference on Information Quality.
More experimental results on blocking x-tuples can be found here.
|
|
Experiment 1: Overall Comparison of Adaptation Strategies
| Movie Database (standard data setting) | (HSQL)
|
|
| DSC1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
|
| |
| |
Experiment 2: Robustness against a varying Dirtiness of the Source Data
| Movie Database | (HSQL)
|
|
| DSH1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.72
|
| |
| |
| Movie Database | (HSQL)
|
|
| DSF1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternative per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.77
|
| |
| |
| Movie Database | (HSQL)
|
|
| DSE1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.81
|
| |
| |
| Movie Database (standard data setting) | (HSQL)
|
|
| DSC1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
| Movie Database | (HSQL)
|
|
| DSA1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.9
|
| |
| |
| Movie Database | (HSQL)
|
|
| DSI1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.933
|
| |
| |
|
| |
| |
Experiment 3: Robustness against a varying Data Uncertainty
| Movie Database (standard data setting) with maximal 10 Alternatives per x-tuple | (HSQL)
|
|
| DSC1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
| Movie Database (standard data setting) with maximal 15 Alternatives per x-tuple | (HSQL)
|
|
| DSC1_15A [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 812,272
|
| Maximal Number of Alternatives per X-Tuple: | 15
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
| Movie Database (standard data setting) with maximal 20 Alternatives per x-tuple | (HSQL)
|
|
| DSC1_20A [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 1,057,191
|
| Maximal Number of Alternatives per X-Tuple: | 20
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
| Movie Database (standard data setting) with maximal 25 Alternatives per x-tuple | (HSQL)
|
|
| DSC1_25A [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 1,301,207
|
| Maximal Number of Alternatives per X-Tuple: | 25
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
|
| |
| |
Experiment 4: Uncertain Keys First
| Movie Database (standard data setting) with maximal 10 Alternatives per x-tuple | (HSQL)
|
|
| DSC1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
| Movie Database (standard data setting) with maximal 15 Alternatives per x-tuple | (HSQL)
|
|
| DSC1_15A [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 812,272
|
| Maximal Number of Alternatives per X-Tuple: | 15
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
| Movie Database (standard data setting) with maximal 20 Alternatives per x-tuple | (HSQL)
|
|
| DSC1_20A [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 1,057,191
|
| Maximal Number of Alternatives per X-Tuple: | 20
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
| Movie Database (standard data setting) with maximal 25 Alternatives per x-tuple | (HSQL)
|
|
| DSC1_25A [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 1,301,207
|
| Maximal Number of Alternatives per X-Tuple: | 25
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
|
| |
| |
Experiment 5: Overall Comparison using Different Blocking Techniques
| Movie Database | (HSQL)
|
|
| DSH1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.72
|
| |
| |
| Movie Database | (HSQL)
|
|
| DSF1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternative per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.77
|
| |
| |
| Movie Database | (HSQL)
|
|
| DSE1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.81
|
| |
| |
| Movie Database (standard data setting) | (HSQL)
|
|
| DSC1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.856
|
| |
| |
| Movie Database | (HSQL)
|
|
| DSA1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.9
|
| |
| |
| Movie Database | (HSQL)
|
|
| DSI1 [zip] |
|
|
| Characteristics: |
|
| Number of X-Tuples: | 102,692
|
| Total Number of Alternatives: | 561,025
|
| Maximal Number of Alternatives per X-Tuple: | 10
|
| Number of Duplicates: | 4,380
|
| Distribution of Clustersizes (clustersize,frequence): | 2,1560; 3,232; 4,72; 5,31; 6,15; 7,11; 8,9; 9,5; 10,3; 11,2; 12,1; 13,1; 15,1
|
| Average Similarity of True Duplicates (scored with Monge-Elkan distance): | 0.933
|
| |
| |
|
| |
| |
| | | | |
