|
1 // Preliminary Part about Code Similarity |
|
2 //======================================== |
|
3 |
|
4 object CW7a { |
|
5 |
|
6 |
|
7 //(1) Complete the clean function below. It should find |
|
8 // all words in a string using the regular expression |
|
9 // \w+ and the library function |
|
10 // |
|
11 // some_regex.findAllIn(some_string) |
|
12 // |
|
13 // The words should be Returned as a list of strings. |
|
14 |
|
15 |
|
16 //def clean(s: String) : List[String] = ... |
|
17 def clean(s: String) : List[String] = |
|
18 "\\w+".r.findAllIn(s).toList |
|
19 |
|
20 //(2) The function occurrences calculates the number of times |
|
21 // strings occur in a list of strings. These occurrences should |
|
22 // be calculated as a Map from strings to integers. |
|
23 |
|
24 |
|
25 //def occurrences(xs: List[String]): Map[String, Int] = .. |
|
26 def occurrences(xs: List[String]) : Map[String, Int] = |
|
27 xs.groupBy(identity).view.mapValues(_.size).toMap |
|
28 |
|
29 //(3) This functions calculates the dot-product of two documents |
|
30 // (list of strings). For this it calculates the occurrence |
|
31 // maps from (2) and then multiplies the corresponding occurrences. |
|
32 // If a string does not occur in a document, the product is zero. |
|
33 // The function finally sums up all products. |
|
34 |
|
35 |
|
36 //def prod(lst1: List[String], lst2: List[String]) : Int = .. |
|
37 def prod(lst1: List[String], lst2: List[String]) : Int = |
|
38 occurrences(lst1).map(x => occurrences(lst2).getOrElse(x._1, 0) * x._2).reduce(_ + _) |
|
39 |
|
40 //(4) Complete the functions overlap and similarity. The overlap of |
|
41 // two documents is calculated by the formula given in the assignment |
|
42 // description. The similarity of two strings is given by the overlap |
|
43 // of the cleaned strings (see (1)). |
|
44 |
|
45 |
|
46 //def overlap(lst1: List[String], lst2: List[String]) : Double = ... |
|
47 def overlap(lst1: List[String], lst2: List[String]) : Double = |
|
48 prod(lst1, lst2).toDouble/Math.max(prod(lst1, lst1).toDouble, prod(lst2, lst2).toDouble) |
|
49 //def similarity(s1: String, s2: String) : Double = ... |
|
50 def similarity(s1: String, s2: String) : Double = |
|
51 overlap(clean(s1), clean(s2)) |
|
52 |
|
53 |
|
54 /* Test cases |
|
55 import CW7a._ |
|
56 val list1 = List("a", "b", "b", "c", "d") |
|
57 val list2 = List("d", "b", "d", "b", "d") |
|
58 occurrences(List("a", "b", "b", "c", "d")) |
|
59 occurrences(List("d", "b", "d", "b", "d")) |
|
60 prod(list1,list2) // 7 |
|
61 overlap(list1, list2) // 0.5384615384615384 |
|
62 overlap(list2, list1) // 0.5384615384615384 |
|
63 overlap(list1, list1) // 1.0 |
|
64 overlap(list2, list2) // 1.0 |
|
65 // Plagiarism examples from |
|
66 // https://desales.libguides.com/avoidingplagiarism/examples |
|
67 val orig1 = """There is a strong market demand for eco-tourism in |
|
68 Australia. Its rich and diverse natural heritage ensures Australia's |
|
69 capacity to attract international ecotourists and gives Australia a |
|
70 comparative advantage in the highly competitive tourism industry.""" |
|
71 val plag1 = """There is a high market demand for eco-tourism in |
|
72 Australia. Australia has a comparative advantage in the highly |
|
73 competitive tourism industry due to its rich and varied natural |
|
74 heritage which ensures Australia's capacity to attract international |
|
75 ecotourists.""" |
|
76 similarity(orig1, plag1) // 0.8679245283018868 |
|
77 // Plagiarism examples from |
|
78 // https://www.utc.edu/library/help/tutorials/plagiarism/examples-of-plagiarism.php |
|
79 val orig2 = """No oil spill is entirely benign. Depending on timing and |
|
80 location, even a relatively minor spill can cause significant harm to |
|
81 individual organisms and entire populations. Oil spills can cause |
|
82 impacts over a range of time scales, from days to years, or even |
|
83 decades for certain spills. Impacts are typically divided into acute |
|
84 (short-term) and chronic (long-term) effects. Both types are part of a |
|
85 complicated and often controversial equation that is addressed after |
|
86 an oil spill: ecosystem recovery.""" |
|
87 val plag2 = """There is no such thing as a "good" oil spill. If the |
|
88 time and place are just right, even a small oil spill can cause damage |
|
89 to sensitive ecosystems. Further, spills can cause harm days, months, |
|
90 years, or even decades after they occur. Because of this, spills are |
|
91 usually broken into short-term (acute) and long-term (chronic) |
|
92 effects. Both of these types of harm must be addressed in ecosystem |
|
93 recovery: a controversial tactic that is often implemented immediately |
|
94 following an oil spill.""" |
|
95 overlap(clean(orig2), clean(plag2)) // 0.728 |
|
96 similarity(orig2, plag2) // 0.728 |
|
97 // The punchline: everything above 0.6 looks suspicious and |
|
98 // should be investigated by staff. |
|
99 */ |
|
100 |
|
101 } |